Zombie pipelines when pipeline monitor restarts

AG-Schumann / Doberman

Distributed supervisory control, data acquisition, and monitoring software for small- to medium-size experiments

MIT License

0 stars 1 forks source link

Zombie pipelines when pipeline monitor restarts #203

Closed adambrown1 closed 1 year ago

adambrown1 commented 1 year ago

the status is still active in the DB but the pipeline is not running, means we cannot restart it through the website

jarongrigat commented 1 year ago

So let's discuss how to solve this issue:

When you stop the hypervisor, the PipelineMonitors stay alive and when you restart the hypervisor, it won't be able to ping the PipelineMonitors and restart them.
In the start-up sequence of the Monitors, they get a list of pipelines of their flavor (alarm, control, convert) that are active or silent
Now the culprit is this line which we added recently to avoid pipelines to be started multiple times (#201)
So my suggestion is to change it to if (doc['status']!='inactive') && (name in self.pipelines.keys()):

adambrown1 commented 1 year ago

What happens if for some reason a pipeline has status inactive in the database but is not actually running? Should we avoid using the database status to make any decisions in order to prevent people from messing things up by changing the database entry to something crazy? But in most cases this looks good. The change from status == active to status != inactive is important in order to avoid multiple instances of pipelines starting when they get silenced.

jarongrigat commented 1 year ago

What we could do is adding a field in the experiment_config for each pipeline monitor that works similar to the processes.managed field for the hypervisor and remove the status field from the individual pipelines.

adambrown1 commented 1 year ago

Sounds reasonable to me, as long as it doesn't overcomplicate filling the pipeline page of doberview