datahandlinglibs - Generic data handling software and utilities

Generic implementation for DUNE DAQ data handling, used in readout, trigger, hsi, ....

Building and setting up the workarea

How to clone and build DUNE DAQ packages, including datahandlinglibs, is covered in the daq-buildtools instructions. For some sample examples that demonstrate how to use datahandlinglibs check out the documentation of datahandlinglibs.

A Deeper Look Into Readout: Functional Elements

Data-flow Diagram (DFD)

The functional elements are seen on the following DFD. Color codes indicate the ownership (or responsibility) of the DAQ subsystems: Dataflow (=blue) and Upstream DAQ (=red).

Domains

Individual domains represent a substantially different path of the incoming data, including some sort of data interpretation, transformation, formatting, or buffering. Functional elements marked with italicized text.

1. Data processing domain (Input: data, Output: processing results): The readout is responsible for generating trigger primitives from raw data, and format these TPs to the agreed data-format. After this, the TPs can be buffered and streamed for other subsystems (most importantly, to Dataselection).
2. Raw Streaming domain (Input: data, Output: error/calib stream): The DAQ needs to interpret incoming data and find possible problems and errors with and within data (e.g.: timestamp continuity violation, data integrity, invalid headers, front-end specific error flags). Calibration flags in form of headers are also inside the front-end data frames. In case these flags are found, some data need to be formatted (e.g.: expanded based on channels ) then streamed to a configured destination (e.g.: local raw binary files or appfwk queues).
3. Requested Data domain (Input: data and data requests -dfmessages::DataRequest-, Output: special data requests to other functional elements and requested data -daqdataformats::Fragment-): This domains contains the conventional "triggered" readout mode. Requested data are extracted from the latency buffers and routed to the appropriate destinations. Special requests (e.g.: recording) may be routed to different domains' functional elements, and data leaving the buffer may be intercepted if needed (e.g.: stream to store).
4. Recorded Data domain (Input: data, "record" requests, Output: recorded data, metadata of data store, transfer acknowledgements and notifications): In case of a record(O(seconds)) request, data leaving the latency buffer are streamed to a transient data store, which is usually local to the readout unit. The recorded data are transferred to other DAQ subsystems with the help of additional metadata, notifications, and acknowledgements.

Definitions

1. Latency Buffer: A container that temporarily stores data, and has certain attributes that ensures search-ability based on a lookup criteria. A notable example for this, is the lookup based on the timestamp, where the timestamp can be converted to an exact position in the buffer if the "timestamp continuity" attribute is ensured in the buffer.

Looking into the directories

At the top level, the datahandlinglibs package uses the same directory structure as other DUNE DAQ packages as described in the daq-cmake documentation. However, due its large number of files, additional subdirectories have been added to organize them. This approach is covered here.