search

dmwm / WMCore

Core workflow management components for CMS.
Apache License 2.0
46 stars 107 forks source link

MSOutput: stop output placement for relval workflows by default #12090

Closed khurtado closed 2 months ago

khurtado commented 2 months ago

Fixes #12069

Status

not-tested

Description

As requested by P&R, add configuration to stop output placement for relval workflows

Is it backward compatible (if not, which system it affects?)

YES

Related PRs

Documentation: https://gitlab.cern.ch/dmwm/wmcore-docs/-/merge_requests/50/diffs

External dependencies / deployment changes

Service configuration changes: https://gitlab.cern.ch/cmsweb-k8s/services_config/-/merge_requests/307 https://gitlab.cern.ch/cmsweb-k8s/services_config/-/merge_requests/308 https://gitlab.cern.ch/cmsweb-k8s/services_config/-/merge_requests/309

khurtado commented 2 months ago

@amaltaro This should skip relval workflow (they would not be uploaded to MongoDB, hence not processed by the MSOutput consumer).

I made it configurable with True as default. Would this configuration be enough? Can we use testbed MSOutput to test?

cmsdmwmbot commented 2 months ago

Jenkins results:

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15182/artifact/artifacts/PullRequestReport.html

amaltaro commented 2 months ago

@khurtado Kenyi, I feel like we should keep producing output documents (persisted in MongoDB) for RelVal, but just don't perform any actual Rucio rule creation if configured so.

About the service configuration, perhaps we could make it more inline with the Tape one: https://gitlab.cern.ch/cmsweb-k8s/services_config/-/blob/prod/reqmgr2ms-output/config-output.py#L78 and call it something like enableRelValDisk.

Note that such RelVal documents would still have to be marked as TransferStatus=done, by MSOutput. Here is one example MSOutput document, despite it having actual rucio rules: https://cmsweb.cern.ch:8443/ms-output/data/info?request=cmsunified_task_BPH-RunIISummer20UL16GEN-00197__v1_T_231129_095358_1095

khurtado commented 2 months ago

@amaltaro Ok, I changed things in a way rucio disk/tape subscriptions are skipped but transfers are marked as done in the MongoDB workflow. Was this what you meant?

cmsdmwmbot commented 2 months ago

Jenkins results:

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15186/artifact/artifacts/PullRequestReport.html

cmsdmwmbot commented 2 months ago

Jenkins results:

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15185/artifact/artifacts/PullRequestReport.html

cmsdmwmbot commented 2 months ago

Jenkins results:

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15188/artifact/artifacts/PullRequestReport.html

khurtado commented 2 months ago

@amaltaro Thanks for the feedback! I overlooked some methods in the pipeline and forced things too soon. I simplified the code and made it more in line with other setting decisions. This should preserve everything else in the pipeline.

EDIT: Here is the updated documentation describing this setting as well. https://gitlab.cern.ch/dmwm/wmcore-docs/-/merge_requests/50

I can also add this parementer in the K8s repository if we move forward with this setting name.

cmsdmwmbot commented 2 months ago

Jenkins results:

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15189/artifact/artifacts/PullRequestReport.html

amaltaro commented 2 months ago

Beautiful! Even better than what I initially thought, thanks!

@khurtado before approving this PR, can you please create the relevant services_config MRs (for test, preprod and prod branches) by explicitly defining this parameter and by default setting it to false?

In addition, please update the documentation with this new hook/detail.

khurtado commented 2 months ago

@amaltaro

Here are the merge request changes for the services config on all 3 branches: https://gitlab.cern.ch/cmsweb-k8s/services_config/-/merge_requests/307 https://gitlab.cern.ch/cmsweb-k8s/services_config/-/merge_requests/308 https://gitlab.cern.ch/cmsweb-k8s/services_config/-/merge_requests/309

And documentation: https://gitlab.cern.ch/dmwm/wmcore-docs/-/merge_requests/50/diffs