export environment variables with a "bot: build" comment

[smoors]

the goal is to have a flexible way to customize a build in a PR comment for one-off situations. the solution proposed here is to add support for exporting environment variables into the job (via sbatch --export), for example:

bot: build export:ENVAR1=value1,ENVAR2=value2

my main use case is to force a rebuild on our cluster (e.g. via EASYBUILD_REBUILD=true) for software that has already been installed previously, but i have several other use cases in mind as well.

any thoughts on this proposal?

[trz42]

For EESSI, specifically the 2023.06 production software stack, we solve such issues by putting easystack (re)build files in sub-directory and process them differently than other easystack files.

When we build for EESSI we run in a shell provided by the compatibility layer by executing /cvmfs/software.eessi.io/versions/2023.06/compat/$(uname -m)/startprefix and this retains only a few environment variables. Instead of setting environment variables in the job (before running startprefix) we put important information into a job.cfg file.

I guess in your case you don't run in a shell of the compatibility layer, hence you don't have that problem.

If we could spin your idea a bit further, say one could add any argument like,

--export=ENVAR1:value1,ENVAR2=value2

one could use that also for additional one-off cases:

• shorter/longer walltime limit (--time=timelimit),
• different memory (--mem=...),
• requesting a GPU (--gpu=...)

Downside would be that it would only work with Slurm (maybe acceptable for the time being). One might want to implement some kind of white/blacklist for what sbatch arguments can be used/not used.

[smoors]

extending this to any job option would make it even more useful indeed. maybe we can use a special keyword, say submit_options, to allow supporting other schedulers in the future?

bot: build submit_options:"--export=ENVAR1=value1,ENVAR2=value2 --time=24:0:0 --mem=10G"

One might want to implement some kind of white/blacklist for what sbatch arguments can be used/not used.

that seems sensible, yes.

I guess in your case you don't run in a shell of the compatibility layer, hence you don't have that problem.

indeed, we don't :)

[boegel]

To control environment variables for the build job, maybe a more direct approach is better:

bot build ... build_env:"EASYBUILD_REBUILD=1,EASYBUILD_HIDDEN=1"

This would then make the bot create a build_env script in the job working directory, which is sourced before the bot/build.sh script is called:

export EASYBUILD_REBUILD=1
export EASYBUILD_HIDDEN=1

source build_env && bot/build.sh

This may make more sense mainly because $SLURM_EXPORT_ENV may also be defined to control which environment variables are passed down into jobs, and when specifying --export we would bypass that (which we probably don't want).