readoutlibs - Generic Readout software and utilities

Generic implementation for DUNE Upstream DAQ Readout Software.

Building and setting up the workarea

How to clone and build DUNE DAQ packages, including readoutlibs, is covered in the daq-buildtools instructions. For some sample examples that demonstrate how to use readoutlibs check out the documentation of readoutmodules.

A Deeper Look Into Readout: Functional Elements

Data-flow Diagram (DFD)

The functional elements are seen on the following DFD. Color codes indicate the ownership (or responsibility) of the DAQ subsystems: Dataflow (=blue) and Upstream DAQ (=red).

Domains

Individual domains represent a substantially different path of the raw data, including some sort of data interpretation, transformation, formatting, or buffering. Functional elements marked with italicized text.

1. Trigger Primitive (TP) handling domain (Input: raw data, Output: TPs): The readout is responsible for generating trigger primitives from raw data, and format these TPs to the agreed data-format. After this, the TPs can be buffered and streamed for other subsystems (most importantly, to Dataselection).
2. Raw Streaming domain (Input: raw data, Output: raw/calib stream): The readout needs to interpret raw data and find possible problems and errors with and withing data (e.g.: timestamp continuity violation, data integrity, invalid headers, front-end specific error flags). Calibration flags in form of headers are also inside the front-end data frames. In case these flags are found, some data need to be formatted (e.g.: expanded based on channels ) then streamed to a configured destination (e.g.: local raw binary files or appfwk queues).
3. Requested Data domain (Input: raw data and data requests -dfmessages::DataRequest-, Output: special data requests to other functional elements and requested data -daqdataformats::Fragment-): This domains contains the conventional "triggered" readout mode. Data from the front-end are processed and routed to the appropriate destinations: either to the Raw Streaming domain (2) or to the raw data buffer (Latency Buffer). Incoming requests are handled via accessing the data buffer then match and extract the requested data for the request. Special requests (e.g.: recording) may be routed to different domains' functional elements, and data leaving the buffer may be intercepted if needed (e.g.: stream to store).
4. Recorded Data domain (Input: raw data, "record" requests, Output: recorded data, metadata of data store, transfer acknowledgements and notifications): In case of a record(O(seconds)) request, data leaving the latency buffer are streamed to a transient data store, which is usually local to the readout unit. The recorded data are transferred to other DAQ subsystems with the help of additional metadata, notifications, and acknowledgements.

Definitions

1. Latency Buffer: A container that temporarily stores the raw data, and has certain attributes that ensures search-ability based on a lookup criteria. A notable example for this, is the lookup based on the timestamp, where the timestamp can be converted to an exact position in the buffer if the "timestamp continuity" attribute is ensured in the buffer.

Class diagram

A zoomable visualization of the readout code for its dunedaq-v2.8.0 release:

Looking into the directories

At the top level, the readout package uses the same directory structure as other DUNE DAQ packages as described in the daq-cmake documentation. However, due its large number of files, additional subdirectories have been added to organize them. This approach is covered here.