When a critical artdaq process dies AND the trace script returns nonzero, DAQInterface hangs

eflumerf commented 2 years ago

This issue has been migrated from https://cdcvs.fnal.gov/redmine/issues/25320 (FNAL account required) Originally created by @jcfreeman2 on 2020-12-14 22:04:40

On December 6th, icarus' run 3894 hung for the following reason:

A boardreader process died
As expected, upon realizing this, DAQInterface entered the recover transition, designed to wind down the artdaq processes as cleanly as possible
Since a trace script was meant to be executed on start, stop and recover (referred to by the DAQINTERFACE_TRACE_SCRIPT variable), DAQInterface called it
For reasons irrelevant to this issue, the trace script returned nonzero, i.e., an error state
DAQInterface responded by throwing an exception which abruptly cancelled the recover transition and was uncaught

The next time (A) a critical process dies, and (B) the trace script returns in an error state, DAQInterface should continue winding down cleanly.

eflumerf commented 2 years ago

Comment by @jcfreeman2 on 2020-12-14 22:43:54

Resolved with feature/25320_handle_process_death_and_trace_error (commit 051abea6c9289f54335f36b940b4b7facca6a1e4)

eflumerf commented 2 years ago

Comment by @wesketchum on 2021-01-15 19:44:26

We've validated that this works in SBN-FD.

art-daq / artdaq_daqinterface