DUNE-DAQ / daq-release

Scripts and configuration files for the DUNE DAQ release
https://dune-daq-sw.readthedocs.io/en/latest/packages/daq-release/
2 stars 0 forks source link

Support multiple software stacks #338

Open jcfreeman2 opened 7 months ago

jcfreeman2 commented 7 months ago

As the DUNE DAQ software project evolves, different parts of it begin to diverge. E.g., writing run control software really isn't in theory heavily coupled with writing detector readout software. Currently we support a near detector stack and a far detector stack, but we should generalize this to more stacks:

To 0th order it seems that these would be in parallel with the ND and FD stacks. E.g., while on this year's April Fool's Day one could call dbt-setup-release or dbt-create for the far detector via NAFD24-04-01, one could also work with the raw data release via NARD24-04-01. Please note that if we want longer, more descriptive names - e.g., NARAWDATA24-04-01 - we'll need to reduce the path length of our releases.

From an implementation perspective, it seems like the strategy to employ would be for each stack to get its own subdirectory in daq-release/configs/ with its own release.yaml file. We could also, which keeping an externals repo and a base package repo in common for all the stacks, have different umbrella packages which pick up different subsets of the packages in those two repos. E.g., we'd rename the current externals umbrella package as externals_fd and externals_nd, and then add an externals_configuration which would drop dpdk but add qt (needed for the dbe package).

jcfreeman2 commented 4 months ago

The first step here seems fairly straightforward, though perhaps I'm underthinking. The most important new umbrella package concerns data formats (I'll call it fddataformats here, though I'm totally open to a new name). As I understand it, the two highest level dependencies for fddataformats are rawdatautils and hdf5libs (*); given that these are already built as part of the fddaq build, it will literally add just a couple of seconds to our nightly build if I create an umbrella package called fddataformats. Then, a developer should be able to do something like:

dbt-create -n NFDDF_PROD4_240415_A9 <name of new work area>
cd <new work area>
dbt-workarea-env

...where FDDF stands for Far Detector Data Formats. Instead of the "traditional" fddaq umbrella package getting loaded, this will spack load in the fddataformats package, even though the hdf5libs and rawdatautils packages will be physically installed in the regular /cvmfs/dunedaq-development.opensciencegrid.org/nightly/NB_PROD4_240415_A9/spack-0.20.0/spack-repo and /cvmfs/dunedaq-development.opensciencegrid.org/nightly/NFD_PROD4_240415_A9/spack-0.20.0/spack-repo repositories, respectively.

(*) For now I'm setting aside the fact that rawdatautils will eventually probably need to be renamed fdrawdatautils and we'll want to add a ndrawdatautils.