[Meta] Create a WMAgent/WMCore-dev image to run unittests

[d-ylee]

Impact of the new feature Create a development image of WMAgent/WMCore that includes unittests and necessary dependencies to run unittests. Unit tests should be fixed if necessary, to adapt to new environment.

Is your feature request related to a problem? Please describe. Related to #11568 , if we want to run unittests in containers, we cannot use the production WMAgent image, since it does not contain unittests.

Describe the solution you'd like Run unit tests in Alma9 with docker. The main changes while doing this are:

• Moving from Centos7 to Alma9
• Running unit tests inside docker container, with bind mounting to /etc, /tmp, /cvmfs, etc
• Moving from Python 3.7 to Python 3.9

Due to the above, there is a number of unit tests failing because of one bullet or another above, which I will be documenting in new issues and linking them to this one (converting this one a meta issue).

• [x] Create Alma9 WMCore-dev image to run unit tests, with CouchDB/MariaDB dependencies
• [x] Add support for EL9 architecture in cmssw scram related unit tests
• [x] Fix Watchdog.IsAlive() for python 3.9
• [ ] https://github.com/dmwm/WMCore/issues/12112
• [ ] https://github.com/dmwm/WMCore/issues/12114
• [ ] https://github.com/dmwm/WMCore/issues/12115
• [ ] https://github.com/dmwm/WMCore/issues/12128

Describe alternatives you've considered

• Similar to the current image for unittests, running all 3 services together: CouchDB + MariaDB + WMAgent

Additional context

[d-ylee]

I have been using a new Alma9 based image to run tests for WMCore. It can be found here: https://github.com/d-ylee/jenkins-test/tree/main/docker/wmcore-dev

[amaltaro]

@d-ylee I don't know if it helps in any shape or form, but yesterday I noticed that the test Alma9 image for MSUnmerged was using AlmaLinux directly instead of Cern Alma9 (I am sure there are some extra packages for the base image, even though I didn't check those).

Based on that, I created a new image to perform other tests and it's available in this commit (it brings in : https://github.com/dmwm/CMSKubernetes/pull/1452/commits/3e32e13ec7c1aa3d01de8e32a9252b9738f6af3d

[d-ylee]

I made a PR for the new image: https://github.com/dmwm/CMSKubernetes/pull/1549

Please take a look!

[khurtado]

@amaltaro @d-ylee I have created 3 new issues and I expect more to come as I look into the unit tests failures in the new environment. Since these are all related to the new WMCore-dev environment, I decided to convert this issue into a meta issue.