search

EESSI / software-layer

Software layer of the EESSI project
https://eessi.github.io/docs/software_layer
GNU General Public License v2.0
20 stars 43 forks source link

use `--from-commit` instead of `--from-pr` for installations done with EasyBuild v4.9.1 #558

Open boegel opened 2 months ago

boegel commented 2 months ago

This doesn't affect the ingested installations at all, but going forward we should use --from-commit instead of --from-pr, and simply use the last commit in the (merged) PR

eessi-bot-aws[bot] commented 2 months ago

Instance eessi-bot-mc-aws is configured to build:

eessi-bot-aws[bot] commented 2 months ago

Instance eessi-bot-mc-azure is configured to build:

bedroge commented 2 months ago

@boegel I'm not entirely sure how this works, but I'm wondering if we shouldn't use the merge commit itself instead of the last commit in the PR? For instance, with the one for Bioconductor you're now using the last commit which actually synced with develop, and using --copy-ec results in downloading the following files:

AMGX-2.4.0_external-thrust.patch
AMGX-2.4.0_fix-openmp-linking.patch
AMGX-2.4.0-foss-2023a-CUDA-12.1.1.eb
arrow-R-11.0.0.3-foss-2022b-R-4.2.2.eb
arrow-R-14.0.1-foss-2023a-R-4.3.2.eb
astropy-5.1.1-intel-2022a.eb
attrdict3-2.0.2-GCCcore-12.3.0.eb
BCFtools-1.19-GCC-13.2.0.eb
CCCL-2.3.0-GCCcore-12.3.0-CUDA-12.1.1.eb
CGAL-5.6.1-GCCcore-13.2.0.eb
cmph-2.0-GCCcore-12.3.0.eb
dm-haiku-0.0.9-foss-2022a-CUDA-11.7.0.eb
dotNET-Core-6.0.420.eb
dotNET-Core-6.0.eb
dotNET-Core-8.0.203.eb
dotNET-Core-8.0.eb
easyconfigs.py
FDMNES-2024-02-29-gomkl-2023a.eb
Giza-1.4.1-GCCcore-13.2.0.eb
gmsh-4.12.2-foss-2023a.eb
h5py-3.11.0-foss-2023b.eb
HTSlib-1.19.1-GCC-13.2.0.eb
ICON-2024.01-foss-2023a.eb
ICON-2024.01_lzacc-undefined.patch
LAPACK-3.12.0-GCC-13.2.0.eb
libfyaml-0.9-GCCcore-12.3.0.eb
libmad-0.15.1b-GCCcore-12.3.0.eb
MATSim-15.0-GCCcore-12.3.0-Java-17.eb
MATSim-15.0-Java-17.eb
MetaMorpheus-1.0.5.eb
METIS-5.1.0-GCCcore-13.2.0.eb
MUMPS-5.6.1-gomkl-2023a-metis-seq.eb
MUMPS-5.6.2_shared-pord.patch
nanomath-1.3.0-foss-2022b.eb
Optax-0.1.7-foss-2022a-CUDA-11.7.0.eb
orthAgogue-20141105-gompi-2023a.eb
PhyML-3.3.20220408-foss-2023a.eb
PhyML-3.3.20220408_mcmc.c-fix.patch
plc-3.10-intel-2022a.eb
plyfile-1.0.3_use_pdm_backend.patch
PnetCDF-1.13.0-iimpi-2023a.eb
POV-Ray-3.7.0.10-GCC-12.3.0.eb
Pysam-0.22.0-GCC-13.2.0.eb
PyTorch-bundle-2.1.2-foss-2023a-CUDA-12.1.1.eb
PyTorch-Geometric-2.5.0-foss-2023a-PyTorch-2.1.2-CUDA-12.1.1.eb
R-bundle-Bioconductor-3.16-foss-2022b-R-4.2.2.eb
R-bundle-Bioconductor-3.18-foss-2023a-R-4.3.2.eb
Rhdf5lib-1.20.0_fix_hardcoded_path_to_mv.patch
SAMtools-1.19.2-GCC-13.2.0.eb
SCOTCH-7.0.4-gompi-2023b.eb
SentencePiece-0.2.0-GCC-12.3.0.eb
SoX-14.4.2-GCCcore-12.3.0.eb
Spack-0.21.2.eb
STAR-2.7.11b-GCC-13.2.0.eb
tbb-2020.3-GCCcore-12.3.0.eb
tiktoken-0.6.0-GCCcore-12.3.0.eb
torchaudio-2.1.2_transform_test_tol.patch
torchaudio-2.1.2_use-external-sox.patch
torch-ignite-0.4.13_dont_destroy_python_path_in_test_launcher.patch
torchtext-0.16.2_download-to-project-root.patch
torchvision-0.16.2_ffmpeg-6.0-fix.patch
torchvision-0.16.2_quantized_tol.patch
VAMPIRE-ASM-6.0-foss-2022b.eb
WGDgc-1.3-foss-2023a-R-4.3.2.eb
xxd-9.1.0307-GCCcore-13.2.0.eb
YACS-0.1.8-GCCcore-12.3.0.eb

The older Bioconductor easyconfigs, which are also updated by that same PR, are not included, is that expected? In other words, what would have happened if our software layer PR would have needed one of those?

The merge commit itself (f8fa6c791dc009696b6cd0280d549163c5b65e38) does return the following files (which is mor ein line with what I expected):

R-bundle-Bioconductor-3.13-foss-2021a-R-4.1.0.eb
R-bundle-Bioconductor-3.14-foss-2021b-R-4.1.2.eb
R-bundle-Bioconductor-3.15-foss-2021b-R-4.2.0.eb
R-bundle-Bioconductor-3.15-foss-2022a-R-4.2.1.eb
R-bundle-Bioconductor-3.16-foss-2022b-R-4.2.2.eb
R-bundle-Bioconductor-3.18-foss-2023a-R-4.3.2.eb

If it does not really matter for the builds themselves, I'm fine with either approach, just wondering how this works.

trz42 commented 2 months ago

In https://github.com/NorESSI/software-layer/pull/344, I first used the merge commit in https://github.com/easybuilders/easybuild-easyconfigs/pull/20379. This failed, because some easyconfig was not available. I then used the last commit before the merge commit. Still the builds failed (maybe for another reason ... have to check).

Edit: Now it fails building the extension Rhdf5lib so probably unrelated to which commit is used.

boegel commented 1 month ago