search

dmwm / WMCore

Core workflow management components for CMS.
Apache License 2.0
46 stars 107 forks source link

Changes required for adoption of the cmsweb-preprod cluster #12067

Open amaltaro opened 2 months ago

amaltaro commented 2 months ago

Impact of the new feature WMCore in general

Is your feature request related to a problem? Please describe. We have recently decided to have a 3rd cluster for integration tests, called cmsweb-preprod. A short summary is: 0) cmsweb-testX clusters: a developer's playground environment to test, patch, change, and do whatever we need when developing for central services. 1) cmsweb-testbed is a level up from the testX clusters. Meant for initial integration tests with more stable code 2) cmsweb-preprod is yet another level up from testbed cluster. Meant for pre-production code, so code that we are actually preparing to push to production. It is meant to be stable! 3) cmsweb-prod/cmsweb is the production environment.

Having said that, we are moving towards the adoption of cmsweb-preprod in CMS (and WM). So this ticket is meant to track all the relevant changes that we need to either a) revise or b) update the codebase for.

Note that some cmsweb-preprod details are still under discussion, e.g. dbs url to be used and separation of CouchDB (and MongoDB) backends.

Describe the solution you'd like 1) https://github.com/dmwm/WMCore/blob/2748054/src/python/WMCore/WorkQueue/DataLocationMapper.py and https://github.com/dmwm/WMCore/blob/27480541849cd97e9058692a6d2ff0601b6ddf86/src/python/WMCore/WorkQueue/Policy/Start/StartPolicyInterface.py#L285 using a code like:

                pileupInstance = '-testbed' if 'cmsweb-testbed' in dbsUrl else '-prod'

2) Create new MongoDB cluster (or database under cmsweb-testbed)

3) Create new CouchDB cluster

4) Considerations for the data stored in the database? I would suggest moving cmsweb-testbed data over to cmsweb-preprod and start with a fresh database in cmsweb-testbed

5) testbed agents will have to move to cmsweb-preprod

In terms of WMCore codebase, I don't think there is anything else to consider. We might want to revisit the workflow json templates though.

Describe alternatives you've considered None

Additional context None

mapellidario commented 3 days ago

After a private discussion with Alan, we agreed that we should start using the new cmsweb-preprod cluster for our final validations and integrations before a deployment to prod. The main blocker for usins cmsweb-preprod consistently at the moment is that cmsweb-preprod shares the same couchdb and mongo databases with cmsweb-testbed

we have two options:

  1. decomission cmsweb-testbed, only use cmsweb-preprod.
    • pro: the need for two separate integration clusters is not clear yet. maybe only use cmsweb-preprod is enough for us
    • cons: many teams rely on cmsweb-testbed, for example cms@home. decomissioning this cluster will require careful communications with other teams.
  2. completely decouple cmsweb-testbed and cmsweb-preprod
    • [ ] ask aroosha to deploy a new couchdb instance in cmsweb-preprod
    • [ ] ask aroosha to deploy a new mongodb cluster for central services running in cmsweb-preprod