search

cms-sw / cmssw

CMS Offline Software
http://cms-sw.github.io/
Apache License 2.0
1.08k stars 4.31k forks source link

[CC7] cmsTriton fails to start container: /lib64/libc.so.6: version `GLIBC_2.28' #45454

Open iarspider opened 3 months ago

iarspider commented 3 months ago

In CMSSW_14_1_X_2024-07-14-0000 for slc7_amd64_gcc12, unit tests and relvals that use Triton are failing:

/cvmfs/oasis.opensciencegrid.org/mis/apptainer/1.3.3/x86_64/bin/apptainer: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by /cvmfs/oasis.opensciencegrid.org/mis/apptainer/1.3.3/x86_64/bin/apptainer)
/cvmfs/oasis.opensciencegrid.org/mis/apptainer/1.3.3/x86_64/bin/apptainer: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by /cvmfs/oasis.opensciencegrid.org/mis/apptainer/1.3.3/x86_64/bin/apptainer)

(full log: link ).

Seems to be caused by update of /cvmfs/oasis.opensciencegrid.org/mis/apptainer/current symlink to a version that doesn't support CC7.

@kpedro88 FYI

iarspider commented 3 months ago

assign heterogeneous

cmsbuild commented 3 months ago

New categories assigned: heterogeneous

@fwyzard,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks

cmsbuild commented 3 months ago

cms-bot internal usage

cmsbuild commented 3 months ago

A new Issue was created by @iarspider.

@Dr15Jones, @antoniovilela, @makortel, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

makortel commented 3 months ago

I'm not too worried on these tests starting to fail on slc7 (that is EOL and we should find a way to be able to drop those builds) as such, but such tests starting to fail independently of any CMS' action sounds worrisome.

kpedro88 commented 3 months ago

I've submitted a ticket to OSG to ask if they can retain support for sl7, but I'm not sure what they'll say.

The use of OSG apptainer was to avoid being frequently stuck with bugs that would break apptainer-in-apptainer until CMS containers were updated and distributed, which is not currently a fast process across all sites, cvmfs repos, etc. I agree this could be improved in several ways:

  1. CMS could distribute its own up-to-date apptainer in a more responsive way
  2. avoiding apptainer-in-apptainer altogether (on the SONIC roadmap, but not achieved yet)
kpedro88 commented 3 months ago

Report from OSG ticket: the apptainer developers do not promise support for sl7 in new versions. OSG recommends pinning to 1.2.5, which is the last version guaranteed to work for all sl7 distributions (1.3.0 potentially broke some things, though up to 1.3.2 had worked in my personal tests). Therefore, I will pin to that version when sl7 is detected.