dmwm / WMCore

Core workflow management components for CMS.
Apache License 2.0
46 stars 107 forks source link

Build a MSUnmerged docker image based on Almalinux #11922

Open amaltaro opened 7 months ago

amaltaro commented 7 months ago

Impact of the new feature MSUnmerged (but we might decide to extend it across WM services)

Is your feature request related to a problem? Please describe. Yes, the gfal2-util package is only maintained in the EPEL RPM repository. Given that our base image is debian based, we cannot fetch updates made to GFAL2 stack, where we rely on third-party packaging/updates to Anaconda repositories. This is the current mechanism to get gfal2 into our MSUnmerged image: https://github.com/dmwm/CMSKubernetes/blob/master/docker/pypi/gfal/Dockerfile

Describe the solution you'd like We have been discussing pros/cons of switching our base image from Debian to what WLCG is adopting (e.g. Almalinux). Many concerns have been expressed with this change though, e.g.: a) image size due to inner-dependency in RPM systems; number of vulnerabilities; and maintenance in general (including the python stack).

With this issue, we would like to prototype a new MSUnmerged docker image based on Almalinux 9 and evaluate it against the 3 concerns listed above, compare it with the current image and consider anything else that might be relevant.

Note that this issue is targeting the MSUnmerged service, which is in the critical path with lack of updates to the GFAL2 stack. Nonetheless, we can also consider updating the WM services to adopt the base Almalinux 9 image, but this will come in the future.

Describe alternatives you've considered None

Additional context None

amaltaro commented 7 months ago

Initial prototype is provided in this PR: https://github.com/dmwm/CMSKubernetes/pull/1452 with very interesting/positive results! I have not tested this service with production payload (e.g. loading different storage plugins), but it seems to be functional.

Current open questions are:

amaltaro commented 5 months ago

This ticket so far depends on the following CMSKubernetes ticket: https://github.com/dmwm/CMSKubernetes/pull/1452

and this WMCore PR - which removes rotatelogs: https://github.com/dmwm/WMCore/pull/11955

Note that the test reqmgr2ms-unmerged:2.3.2rc6-20240419 image no longer depends on rotatelogs.