DUNE-DAQ / daq-release

Scripts and configuration files for the DUNE DAQ release
https://dune-daq-sw.readthedocs.io/en/latest/packages/daq-release/
2 stars 0 forks source link

Support a `fddatautilities` umbrella package #361

Open jcfreeman2 opened 4 months ago

jcfreeman2 commented 4 months ago

...which will depend directly only on hdf5libs and rawdatautils, and can be used for data analysis but isn't a full blown DAQ system the way fddaq and nddaq are.

A major part of this Issue is that we want to have an image available to people who want to use fddatautilities which is much slimmer than the fddaq/nddaq analogue, ghcr.io/dune-daq/nightly-release-alma9:development_v5 and the image which it derives from, ghcr.io/dune-daq/alma9-slim-externals:v2.1, both north of 10 GB.

A secondary effect of this Issue is it should give us a real-world sense of what extending our framework to support diverse types of umbrella package entails; this topic is covered in more detail in Issue #338

jcfreeman2 commented 4 months ago

Status update:

What we have so far:

To get fddatatutilities inside a container based on the above image, you can just do the following:

 . /cvmfs/dunedaq-development.opensciencegrid.org/nightly/NFDDU_DEV_240408_A9/spack-installation/share/spack/setup-env.sh 
spack load fddatautilities

and then run spack spec ..., etc., on it.

As you can see, NFDDU_DEV_240408_A9 is a new repository, separate from the core DUNE DAQ repository or the externals repository. In this sense it's analogous to NFD_DEV_240408_A9. While it is the case that the fddetdataformats and rawdatautils packages in this repository are also found in NFD_DEV_240408_A9, this won't in general be the case for different collections of packages (e.g., run control packages) in the future; thus I made the decision to use a separate repo rather than inserting an fddatautilities umbrella package into the far detector repo.

fddatautilities depends on a variant of dunedaq (*), dunedaq subset=datautilities. Moving forward I believe it makes sense that whenever we have a new umbrella package, we add a new possible value to subset to specify which core dunedaq packages are needed. To extend that point, we likely should do the same thing for the externals repo - currently it's the case that the externals repo in the image is just the usual ext-v2.1 externals repo but with all unnecessary externals deleted out of it. Instead we should have externals subset=datautilities, etc.

One question that will be persistent as we move forward is, how do we decide which packages should go into the core DUNE DAQ repository, and which should go into the umbrella-package-specific repository? There are tradeoffs involved: the choice I made here means that fddetdataformats and rawdatautils are duplicated between the fddatautilities and fddaq repositories, but on the other hand they don't take up a ton of space and it seems like potential overkill to throw any package which might ever be used by more than one umbrella package into the core DUNE DAQ repository.

Another question: I imagine that we want to treat these new umbrella packages as exactly analogous to fddaq and nddaq, correct? In other words, whenever a new umbrella package like fddatautilities gets added it should be possible to pass it to dbt-setup-release and dbt-create?

(*) Work on this Issue began before dunedaq was renamed coredaq a few days ago