search

dmwm / WMCore

Core workflow management components for CMS.
Apache License 2.0
46 stars 107 forks source link

Update Jenkins WMCore baseline and PR-test to run based on WMAgent/CouchDB/MariaDB containers #11568

Open amaltaro opened 1 year ago

amaltaro commented 1 year ago

Impact of the new feature WMCore CI

Is your feature request related to a problem? Please describe. As we move WMAgent deployment model away from RPMs, we need to update Jenkins setup to use the same software stack as we will be running in production.

For further details about our WMCore Jenkins setup and which jobs are executed for pull request tests, please refer to: https://github.com/dmwm/WMCore/wiki/Understanding-Jenkins#tasks-executed-by-the-ci-jenkins-setup

Describe the solution you'd like The final solution is not yet clear to me, as we might want to opt for a single container for WMAgent + CouchDB + MariaDB for our CI setup. If we follow this path, then we should start looking into this repository: https://github.com/dmwm/Docker

which defines how the WMCore CI-based Docker image is constructed and which setup/scripts come with it.

Otherwise, please refer to the alternatives section below.

Describe alternatives you've considered The alternative would be to use the same docker images that we will be using for central production and composing/interconnecting all of them together in the CI setup.

From the RPM era, we have been using a different package named to bring in some development tools (pylint, codestyle, nose, etc) which are likely not available in the default wmagent pypi package, see: https://github.com/cms-sw/cmsdist/blob/comp_gcc630/wmagentpy3-dev.spec

If needed, we will have to create a new PyPi package for it. Or we deploy the extra packages in the Dockerfile.

Additional context For further reference, many of the WMCore Jenkins scripts executed actually come from this repository: https://github.com/cms-sw/cms-bot/tree/master/DMWM

Part of the following meta issue: https://github.com/dmwm/WMCore/issues/11314

vkuznet commented 1 year ago

Alan, could you please provide a reason why should we use your specific repositories. I found that https://github.com/amaltaro/Docker is a fork of https://github.com/dmwm/Docker, and https://github.com/amaltaro/cms-bot/tree/master/DMWM is a clone of https://github.com/cms-sw/cms-bot/tree/master/DMWM. Would it be more appropriate to work with dmwm based repositories, and clone cms-sw one into dmwm umbrella?

amaltaro commented 1 year ago

@vkuznet Valentin, I updated the initial description to actually point to the official repositories (instead of my remote).

Regarding your question, I am not aware of any strong reason on why we should keep using those repositories. In fact, I would be totally in favor of organizing it in another repository that we have more control and that we can have everything that we need for Jenkins.

d-ylee commented 5 months ago

@amaltaro I am currently creating a new WMCore development image that is pulling in the different test scripts from the cmssw and cmsbot repos.

d-ylee commented 5 months ago

Additional updates: I created a WMCore-PR-pylint Pipeline and was able to successfully run PR pylint and grab artifacts. Currently using a hardcoded PR#11995 and TargetBranch Master.

Jenkinsfile: https://github.com/d-ylee/jenkins-test/blob/main/WMCore-PR-pylint/Jenkinsfile Results: https://cmssdt.cern.ch/dmwm-jenkins/view/WMCore%20Docker/job/WMCore-PR-pylint/13/

d-ylee commented 2 weeks ago

We have currently migrated the Repository with the Jenkinsfiles to https://github.com/dmwm/WMCore-Jenkins. All jobs using Jenkins Declarative Pipelines from SCM have been updated with the new repo.