Add setter methods map to WMWorkload and call it for all reqArg parameters

[todor-ivanov]

Fixes #12038

Status

Ready

Description

With the current PR we add the new functionality to call all needed WMWorkload setter methods based on a full reqArgs dictionary passed from the upper level caller and try to set all of parameters at once, rather than calling every single setter method one by one. This is achieved by creating a proper map between methods and possible request arguments. And later validating if the set of arguments passed with reqArgs dictionary would properly match the signature of the setter method which is to be called. In the current implementation only a small set of methods is mapped to request parameters :

• setSiteWhiteList
• setSiteBlacklist
• setPriority (Mostly single argument parametrized methods, but that's ok for the time being, because those cover perfectly the functionality we need to plug in here)

With this change a path for updating all possible workqueue elements in a single go was opened. In the WorkQueue service an additional method was developed for fetching all possible workqueue elements and update them all with the full set of arguments provided through the reqArgs dictionary with the cost of a single database action per WorkQuee element, rather than 3 (or more) separate calls to the database for updating every single element parameter separately. Upon updating all workqueue elements with the new parameters the WMSpec copy of the given workflow at the workqueue is also updated in a single push using the same logic from above.

Is it backward compatible (if not, which system it affects?)

YES

Related PRs

None

External dependencies / deployment changes

None

[cmsdmwmbot]

Jenkins results:

• Python3 Unit tests: succeeded
  • 1 tests added
  • 2 changes in unstable tests
• Python3 Pylint check: failed
  • 10 warnings and errors that must be fixed
  • 9 warnings
  • 195 comments to review
• Pylint py3k check: succeeded
• Pycodestyle check: succeeded
  • 16 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15218/artifact/artifacts/PullRequestReport.html

[cmsdmwmbot]

Jenkins results:

• Python3 Unit tests: failed
  • 2 new failures
• Python3 Pylint check: failed
  • 11 warnings and errors that must be fixed
  • 9 warnings
  • 212 comments to review
• Pylint py3k check: succeeded
• Pycodestyle check: succeeded
  • 19 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15219/artifact/artifacts/PullRequestReport.html

[todor-ivanov]

@amaltaro Please take a look at this PR, I think this one fully covers all the requirements and addresses our fears for affecting scalability due to increased database calls.

[cmsdmwmbot]

Jenkins results:

• Python3 Unit tests: succeeded
  • 1 changes in unstable tests
• Python3 Pylint check: failed
  • 7 warnings and errors that must be fixed
  • 9 warnings
  • 211 comments to review
• Pylint py3k check: succeeded
• Pycodestyle check: succeeded
  • 19 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15220/artifact/artifacts/PullRequestReport.html

[cmsdmwmbot]

Jenkins results:

• Python3 Unit tests: succeeded
  • 1 tests no longer failing
  • 1 changes in unstable tests
• Python3 Pylint check: failed
  • 7 warnings and errors that must be fixed
  • 9 warnings
  • 211 comments to review
• Pylint py3k check: succeeded
• Pycodestyle check: succeeded
  • 19 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15240/artifact/artifacts/PullRequestReport.html

[cmsdmwmbot]

Jenkins results:

• Python3 Unit tests: succeeded
  • 1 tests no longer failing
• Python3 Pylint check: failed
  • 7 warnings and errors that must be fixed
  • 9 warnings
  • 211 comments to review
• Pylint py3k check: succeeded
• Pycodestyle check: succeeded
  • 19 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15241/artifact/artifacts/PullRequestReport.html

[todor-ivanov]

hi @amaltaro I have addressed your comments - Mostly, I removed the WQE status filter, based on the investigation I did on the JobUpdater component. You may take another look.

[cmsdmwmbot]

Jenkins results:

• Python3 Unit tests: failed
  • 1 new failures
• Python3 Pylint check: failed
  • 7 warnings and errors that must be fixed
  • 9 warnings
  • 211 comments to review
• Pylint py3k check: succeeded
• Pycodestyle check: succeeded
  • 19 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/15249/artifact/artifacts/PullRequestReport.html

[cmsdmwmbot]

Can one of the admins verify this patch?

[anpicci]

@todor-ivanov apart from my comment to @amaltaro 's review, I only suggest to resolve the conflicts on src/python/WMCore/ReqMgr/Service/Request.py, if you have not considered yet to address that

[anpicci]

@vkuznet thank you for the feedback. @todor-ivanov @amaltaro , I propose to proceed as follows:

• let's stick with the original Todor's proposal to use maps;
• @amaltaro will focus his review on checking that such implementation is valid and it is consistent with the rest of the system;
• at the same time, @todor-ivanov provides documentation both for the external packages introduced in the PR, and in terms of docstring and in-line comments, when necessary according to both developer and reviewer point of view, such that maintenance of the code is enhanced;
• once Jenkins tests are passed and the PR merged, @todor-ivanov is responsible for fixing issues arising during future release validations that are ascribible to this PR, or, at least, investigate and provide guidances for resolving such issues.

I want to clarify that I would like to adopt these accountability principles to every issue, meaning that this is not a special treatment outlined only for this PR. I will follow up on this during the next group meeting.

Thanks!