Removing/unassigning/inactivating a leader from experiment with a running profile will end the profile

[CamDavidsonPilon]

some notes:

• When a leader runs a profile, the job is saved to it's local job DB with the metadata name="experiment_profile", job_source="user", experiment="<experiment>", unit="<leader>", pid=<number>.
• When a unit:
  • goes inactive, the following runs: pio kill --all-jobs. This kills all jobs, expect those we deem "long-running"
  • gets unassigned, the following runs: pio kill --experiment <exp>, this kills all jobs tagged with experiment="<exp>".
  • gets removed: this can't happen to leader, so moot.

1. What if we make experiment profile as "long running?" Inactive would not kill it, but unassigned would.
2. What if we make experiment profile not associated to an experiment? This would create conflicts if multiple profiles want to run for different experiments. It's also not clear how to kill it (and only it) from the command line: pio kill --name experiment_profile would kill all profiles, and pio kill --name experiment_profile --experiment <exp> would kill nothing.

Probably something along the lines of 1. makes more sense. However, what does kill this profile? pio kill --name experiment_profile --experiment <exp>, or over MQTT (UI). How do we handle unassignment killing it?

[CamDavidsonPilon]

Thinking more about this. How does one end an experiment profile? Either:

1. It ends naturally
2. The user stops it early from the /profiles page
3. The experiment is ended (via the UI).