search

DUNE-DAQ / daq-release

Scripts and configuration files for the DUNE DAQ release
https://dune-daq-sw.readthedocs.io/en/latest/packages/daq-release/
2 stars 0 forks source link

Create a mockup/initial version of release in cvmfs #1

Closed dingp closed 3 years ago

dingp commented 4 years ago

As part of the DAQ release model, we are exploring how to organize a release.

The initial plan is to have separate directories for each DAQ release, in which only the version of packages used in the specific release are included. This will help both developer/user and release manager to get a clearer view of softwares used in certain release and minimize risks of using incompatible versions of a package in a given release.

The following proposed solution is for the near-term usage of UPS as our software packaging technique. It does not apply to other packaging techniques, e.g. spack. (spack view in combination with some homebrew scripts might be a nice solution to achieve similar goals. We will explore that in the future.)

In the context of using UPS products, the release subdirectory can contain symbolic links to the versions of packages used or the actual package contents.

The following directory structure illustrates one proposed setup with symlinks under releases subdirectory to the chosen versions of UPS products under /cvmfs/dune.opensciencegrid.org/DUNE/products.

By default, the UPS products directory a user/developer needs to use is just the release subdirectory, in which only one version for each package is avalilable (i.e. ups list -aK+ shows only one version for each package). Technichally, this is achived by only including the releases/v1.0.0/packages and releases/v1.1.0/externals in the PRODUCT environment variable.

If different versions of packages need to be used for development/test purpose, one can add the /cvmfs/dune.opensciencegrid.org/dunedaq/DUNE/products to the PRODUCT env to get access to those.

/cvmfs/dune.opensciencegrid.org/dunedaq/DUNE/
└── products
│   ├── appfwk
│   │   ├── v1.0.0
│   │   └── v1.0.0.version
│   ├── boost
│   │   ├── v1_70_0
│   │   ├── v1_70_0.version
│   │   ├── v1_73_0
│   │   └── v1_73_0.version
│   ├── ers
│   │   ├── v0_26_00
│   │   └── v0_26_00.version
│   ├── gcc
│   │   ├── v8_2_0
│   │   └── v8_2_0.version
│   ├── setup
│   ├── setups
│   ├── setups_layout
│   ├── setups.new
│   ├── ...
│   ├── ...
│   ├── ...
│   ├── TRACE
│   │   ├── v3_15_09
│   │   └── v3_15_09.version
│   └── ups
│       ├── current.chain
│       ├── v6_0_8
│       └── v6_0_8.version
└── releases
    └── v1.0.0
        ├── externals
        │   ├── boost
        │   │   ├── v1_73_0 -> ../../../../products/boost/v1_73_0
        │   │   └── v1_73_0.version -> ../../../../products/boost/v1_73_0.version/
        │   ├── gcc
        │   │   ├── v8_2_0 -> ../../../../products/gcc/v8_2_0
        │   │   └── v8_2_0.version -> ../../../../products/gcc/v8_2_0.version/
        │   ├── setup
        │   ├── setups
        │   ├── setups_layout
        │   ├── setups.new
        │   ├── ...
        │   ├── ...
        │   ├── ...
        │   ├── TRACE
        │   │   ├── v3_15_09 -> ../../../../products/TRACE/v3_15_09
        │   │   └── v3_15_09.version -> ../../../../products/TRACE/v3_15_09.version/
        │   └── ups
        │       ├── current.chain -> ../../../../products/ups/current.chain
        │       ├── v6_0_8 -> ../../../../products/ups/v6_0_8
        │       └── v6_0_8.version -> ../../../../products/ups/v6_0_8.version/
        └── packages
            ├── appfwk
            │   ├── v1.0.0 -> ../../../../products/appfwk/v1.0.0
            │   └── v1.0.0.version -> ../../../../products/appfwk/v1.0.0.version
            ├── ...
            ├── ...
            ├── ...
            └── ers
                ├── v0_26_00 -> ../../../../products/ers/v0_26_00
                └── v0_26_00.version -> ../../../../products/ers/v0_26_00.version/
dingp commented 4 years ago

The workflow of creating a new release may look like:

  1. Have packages used in a release ready under /cvmfs/dune.opensciencegrid.org/dunedaq/DUNE/products (this step may be a manual step in the beginning, but may later be part of a CI workflow);
  2. A script taking input of a recipe file (release manifest file) will make appropriate new directories under /cvmfs/dune.opensciencegrid.org/dunedaq/DUNE/releases and make symlinks to packages under /cvmfs/dune.opensciencegrid.org/dunedaq/DUNE/products area.
dingp commented 4 years ago

Renamed /cvmfs/dune.opensciencegrid.org/dunedaq/DUNE/releases to /cvmfs/dune.opensciencegrid.org/dunedaq/DUNE/releases-tmp to avoid confusions before we officially announce the release model.