search

easybuilders / easybuild-framework

EasyBuild is a software installation framework in Python that allows you to install software in a structured and robust way.
https://easybuild.io
GNU General Public License v2.0
148 stars 201 forks source link

change default job backend to Slurm + deprecate support for using GC3Pie as job backend #4565

Open boegel opened 2 months ago

gkaf89 commented 1 month ago

It seems that the current implementation of the Slurm backend does not support the experimental easystack feature.

When compiling easystacks compilation jobs are resubmitted without dependencies on previous entries of the easystack. Consider for instance the easystack file:

easyconfigs:
  - foss-2024.05.eb
  - HDF5-1.14.3-gompi-2023b.eb

Jobs for GCCcore are created for both foss-2024.05.eb and HDF5-1.14.3-gompi-2023b.eb, and there is no dependency that prevents the job for HDF5-1.14.3-gompi-2023b.eb to start after the job for foss-2024.05.eb.

I am mentioning the issue here because easystacks seem like a good use case for parallel compilation jobs.

ocaisa commented 1 month ago

@gkaf89 I looked into that in https://github.com/easybuilders/easybuild-framework/issues/4558#issuecomment-2160471934 but I would need to find time to implement

boegel commented 1 month ago

@gkaf89 That's correct, the integration of easystack files and --job is far from optimal, but that's orthogonal to which job backend is being used (so better to continue that discussion in #4558, indeed)