Start planning for transition to EL9 Apptainer default container

[shreyb]

We need to plan for the SL7 EOL. The default Apptainer image is currently an SL7 one:

https://github.com/fermitools/jobsub_lite/blob/f5bf8b78cc286f45bb54e2ea3e4336645e5a67cf/lib/utils.py#L41

In consultation with experiments, we need to change this to AL9:

/cvmfs/singularity.opensciencegrid.org/fermilab/fnal-wn-el9:latest

• [x] Discuss plan with jobsub project group, come up with transition idea and rough timeline
• [x] Discuss with experiments at FIFE meeting - what would their preferred timeline be? Can we do this BEFORE SL7 EOL?
• [x] Document the timeline here, implement it (it should just be that one line change), including testing
• [x] Send announcement to stakeholders
• [x] Implement change, including testing

[shreyb]

Management has told us that this needs to happen sooner rather than later. The plan is to release this in March 2024 as 1.7.

[shreyb]

One idea - perhaps add this to the list of environment variables that jobsub inspects. JOBSUB_DEFAULT_SINGULARITY_IMAGE perhaps.

[marcmengel]

That would move it to the config. And I like that people could change it in, say, an experiment setup script if their default is something different.

[shreyb]

So I was thinking about not even having it in the config repo at all. Basically, change the hardcoding in utils.py to be:

DEFAULT_SINGULARITY_IMAGE = os.environ.get("JOBSUB_DEFAULT_SINGULARITY_IMAGE",
    "/cvmfs/singularity.opensciencegrid.org/fermilab/fnal-wn-el9:latest"
)

And just document how users can override that default. Then we wouldn't have to touch the config.

How does that sound?

[marcmengel]

That works.

[shreyb]

This change has gone out in 1.7-rc1, and will be included in 1.7. Closing issue.