search

AG-Schumann / Doberman

Distributed supervisory control, data acquisition, and monitoring software for small- to medium-size experiments
MIT License
0 stars 1 forks source link

Pipelines can be started multiple times in one monitor #196

Closed adambrown1 closed 1 year ago

adambrown1 commented 1 year ago

The pipeline monitor doesn't seem to check if a pipeline is already running before starting it. That means if you send the start command twice (easy to do from the website, where it only moves up to "active" with some delay) it runs twice in parallel and is then "unstoppable" (without restarting the monitor).

Can probably be fixed by adding an if after line 34 in start_pipeline, to check whether p.name is in self.pipelines or whether its status is set to active in the database. Or both -- what should we do if they disagree?

https://github.com/AG-Schumann/Doberman/blob/fb0bca14be49db0f2bbe0c919b0778cb3af6c36d/Doberman/PipelineMonitor.py#L31-L35

Optionally, we could set the DB status to active early in the process before constructing the Pipeline, and reset to inactive in the except clause of L41 if it failed to start, to make this even less likely to happen.

Unrelated, the return 1, return 0 seems never to be checked and anyway isn't very pythonic. Could get rid of it.

adambrown1 commented 1 year ago

@TiffanyLuce Are you interested in looking into this, or do you have other things to do?

TiffanyLuce commented 1 year ago

It seems to be a good exercise to flirt with the code and understand it better so I'm willing to look into it !

adambrown1 commented 1 year ago

If pipelines used to control other pipelines this ends up resulting in an ever-increasing number of pipelines, and alarms stops working -> have tagged as critical

jarongrigat commented 1 year ago

Should we close this issue? Also #197 ?

adambrown1 commented 1 year ago

We had a problem last week that seemed to be a similar symptom, even with Tiffany's changes. Can I test it for a few more days before we claim it is solved?