Open jcfreeman2 opened 4 months ago
Status update:
What we have so far:
ghcr.io/dune-daq/alma9-slim-externals:fddatautilities
, an image which contains only the externals needed by fddatautilities
and which is 6.8 GB as opposed to 9.1 GB for ghcr.io/dune-daq/alma9-slim-externals:development_v5
ghcr.io/dune-daq/nightly-release-alma9:fddatautilities
, which contains the fddatautilities
umbrella packageTo get fddatatutilities
inside a container based on the above image, you can just do the following:
. /cvmfs/dunedaq-development.opensciencegrid.org/nightly/NFDDU_DEV_240408_A9/spack-installation/share/spack/setup-env.sh
spack load fddatautilities
and then run spack spec ...
, etc., on it.
As you can see, NFDDU_DEV_240408_A9
is a new repository, separate from the core DUNE DAQ repository or the externals repository. In this sense it's analogous to NFD_DEV_240408_A9
. While it is the case that the fddetdataformats
and rawdatautils
packages in this repository are also found in NFD_DEV_240408_A9
, this won't in general be the case for different collections of packages (e.g., run control packages) in the future; thus I made the decision to use a separate repo rather than inserting an fddatautilities
umbrella package into the far detector repo.
fddatautilities
depends on a variant of dunedaq
(*), dunedaq subset=datautilities
. Moving forward I believe it makes sense that whenever we have a new umbrella package, we add a new possible value to subset
to specify which core dunedaq packages are needed. To extend that point, we likely should do the same thing for the externals repo - currently it's the case that the externals repo in the image is just the usual ext-v2.1
externals repo but with all unnecessary externals deleted out of it. Instead we should have externals subset=datautilities
, etc.
One question that will be persistent as we move forward is, how do we decide which packages should go into the core DUNE DAQ repository, and which should go into the umbrella-package-specific repository? There are tradeoffs involved: the choice I made here means that fddetdataformats
and rawdatautils
are duplicated between the fddatautilities
and fddaq
repositories, but on the other hand they don't take up a ton of space and it seems like potential overkill to throw any package which might ever be used by more than one umbrella package into the core DUNE DAQ repository.
Another question: I imagine that we want to treat these new umbrella packages as exactly analogous to fddaq
and nddaq
, correct? In other words, whenever a new umbrella package like fddatautilities
gets added it should be possible to pass it to dbt-setup-release
and dbt-create
?
(*) Work on this Issue began before dunedaq
was renamed coredaq
a few days ago
...which will depend directly only on
hdf5libs
andrawdatautils
, and can be used for data analysis but isn't a full blown DAQ system the wayfddaq
andnddaq
are.A major part of this Issue is that we want to have an image available to people who want to use
fddatautilities
which is much slimmer than thefddaq
/nddaq
analogue,ghcr.io/dune-daq/nightly-release-alma9:development_v5
and the image which it derives from,ghcr.io/dune-daq/alma9-slim-externals:v2.1
, both north of 10 GB.A secondary effect of this Issue is it should give us a real-world sense of what extending our framework to support diverse types of umbrella package entails; this topic is covered in more detail in Issue #338