search

EESSI / filesystem-layer

Filesystem layer of the EESSI project
https://eessi.github.io/docs/filesystem_layer
GNU General Public License v2.0
7 stars 16 forks source link

support ingesting of combined tarballs for software layer #142

Open boegel opened 1 year ago

boegel commented 1 year ago

It would be interesting if the ingest-tarball.sh script would support ingesting a combined tarball of software installations for the software layer, in which a small number of installations across all (or at least multiple) CPU targets are included.

For OSU-Micro-Benchmarks (see PR #201), we had to create and ingest 10 separate small tarballs, which only adds overhead (since for each tarball, a PR to the staging repo is created, and needs to be reviewed/merged). The bot could combine all tarballs into a single one when it's instructed to do the deploy.

The ingest script currently makes a couple of assumptions that are incorrect for such a combined tarball, including that the tarball only includes software installations for a single CPU target, and that tar tf ${tarball} | head -1 allows to determine whether it's an ingesting for software/compat/init .

boegel commented 4 months ago

A problem arose today when trying to ingest a tarball that included updates hooks (2023.06/init/easybuild/eb_hooks.py), since that messed up the way in which the CPU architecture is being checked, which is based on this:

tar_first_file=$(tar tf "${tar_file}" | head -n 1)

@bedroge Do you think it's OK if we add grep -v '/init/' in there, before the head? I guess we should also make sure that $tar_first_file is not empty then...