SainsburyWellcomeCentre / aeon_experiments

Experiment workflows for Project Aeon
BSD 3-Clause "New" or "Revised" License
2 stars 0 forks source link

Add watchdog monitor for auto-restarting ephys workflows #543

Closed glopesdev closed 4 months ago

glopesdev commented 4 months ago

The ONIX harp sync timeout notification depended on its own stream for the timestamp and therefore was never sent out. We decided to use a system timestamp instead so at least some notification is sent.

We have observed sporadic crashes due to tether and other physical hardware issues. Due to the critical nature of the rotary commutator to ensure stability of experiments, we decided to have an auto-restart loop with a short timer on the launch script which will attempt to restart the workflow automatically.

Another issue we have encountered related to tether and connection problems is on occasion the probe ephys stream might simply stop. We now report this as an error and crash the workflow so the auto-restart loop can act through a single pathway.

Fixes #541