search

UCBerkeleySETI / commensal-automator

Automation for Breakthrough Listen's commensal observing
MIT License
1 stars 1 forks source link

hpguppi_proc timeout leaves us in "processing" state #3

Closed lacker closed 2 years ago

lacker commented 2 years ago

I think this will leave the automator wedged so that it can't do anything any more?

[2022-10-18 07:23:37,478 - INFO - proc_hpguppi.py:69] Processing file: guppi_59867_42817_000358_J1830-3602_0001.0000.raw
[2022-10-18 07:27:52,555 - INFO - automator.py:198] New state: not-tracking
[2022-10-18 07:29:38,426 - INFO - automator.py:198] New state: tracking
[2022-10-18 07:29:38,426 - INFO - automator.py:326] Processing still in progress, therefore not recording current track.
[2022-10-18 07:31:42,562 - INFO - automator.py:198] New state: not-tracking
[2022-10-18 07:31:42,562 - INFO - automator.py:380] No active recording for endof track for array_1
[2022-10-18 07:32:02,162 - INFO - automator.py:198] New state: tracking
[2022-10-18 07:32:02,162 - INFO - automator.py:326] Processing still in progress, therefore not recording current track.
[2022-10-18 07:33:37,548 - ERROR - proc_hpguppi.py:124] Timeout of 600 seconds exceeded
[2022-10-18 07:33:37,548 - ERROR - proc_hpguppi.py:74] Timed out, processing has not started.
[2022-10-18 07:33:37,548 - ERROR - automator.py:432] hpguppi_proc timed out. Stopping automator for debugging.
[2022-10-18 08:02:06,161 - INFO - automator.py:198] New state: not-tracking
[2022-10-18 08:02:06,161 - INFO - automator.py:380] No active recording for endof track for array_1
[2022-10-18 08:02:24,961 - INFO - automator.py:198] New state: tracking
[2022-10-18 08:02:24,961 - INFO - automator.py:326] Processing still in progress, therefore not recording current track.
lacker commented 2 years ago

I changed it so that it goes into "paused" mode, where it keeps running but does nothing on new messages. this still wedges the automator, but it's intentionally wedging the automator, so that we can debug if need be. to get out of pause, you just have to restart. https://github.com/UCBerkeleySETI/commensal-automator/commit/524a9cd1f7e545c4f7aacc4a51bbe5b0f7a91b91