search

DUNE-DAQ / dfmodules

Dataflow applications
3 stars 1 forks source link

tp-stream-writer not adapted to multiple TPSet producers? #384

Open glehmannmiotto opened 1 month ago

glehmannmiotto commented 1 month ago

When there is more than 1 TPSet producer (e.g. 3 per each detector unit, one per plane) it seems that the TP writer logic is not taking into account the fact that TPSets won't come in strictly ordered time. Therefore, we get warnings like Unable to create the dataset "TR_Builder_0x00000004_TimeSliceHeader": (Links) Object already exists meaning (if I understand correctly) that TPs are coming in for the time slice that was already closed. In the case of multiple streams this is probably to be expected, except if there is a (configurable) grace time in which a slice is kept open while the next is being filled already. Is this analysis correct?

bieryAtFnal commented 1 month ago

In both the v4 and the v5 code, there is a grace period in the TPStreamWriter to allow TPs within a given TimeSlice to arrive independently of one another. Actually, the logic is even more forgiving than that. What it uses is a 'quiet time' of a specified length, and only when no TPs have been received for that length of time is a given TimeSlice closed and written out.

In the v5 code, that interval is hard-coded to 1 second. So, this means that when we see warning messages such as what you describe, TPs from different sources are arriving more than 1 second apart.

In the v4 code, the quiet-time interval is configurable. And, there are other enhancements, such as the ability to turn off the warning log messages when tardy TPs arrive and the addition of metrics that indicate how many TPs are lost for this reason and how big the interval is between the initial and subsequent groups of TPs. These changes won't prevent a DAQ session from producing tardy TPs, but they can provide better insight into how far out of sync various TP generators have become in a given data-taking run.

In the v5.2 release preparation discussions, porting these v4 enhancements to v5 was deemed less important than other changes that are being developed for the release, so that other work has taken priority. But, if these TPStreamWriter changes are strongly desired for v5.2, we can talk about trying to get them included.