Split up Reco workflow creation

[sfoulkes]

We discussed at Bristol breaking up the workflow creation so that the person making a request only specifies processing parameters and dataops can control other parameters at approval time. Going off the white board, the following parameters are available to the person making the request:

• User/Group
• Request Priority
• DBS URL
• CMSSW Version
• Global Tag
• Input Dataset
• Run/Block White/Black Lists
• Processing ConfigCache ID/Config.DataProcessing Scenario
• Multiple skims, for each skim:
  • ConfigCache ID/Config.DataProcessing Scenario and skim name

After approval DataOps would be able to set the following:

• Team/Agent
• Priority
• Site White/Black List
• Merged and Unmerged LFN base
• Job Splitting algos for each processing and skim task
• Acquisition Era
• Processing Version
• Merge Parameters
• Retry Policy

We need to make changes to the Workload helpers to facilitate this and make the ReReco spec tolerant of this. We also need to modify the ReqMgr for the new workflow.

[evansde77]

evansde: The list of parameters looks good for both lists, I wonder if we want to have two cases:

1. Official ReReco - Stuff from Conf/DataProcessing scenario configs (skims from that as well) benefit being that the API becomes really really simple
2. DataProcessing request - generic one step processing with config cache URL. Can be used for reco, extra skims etc. Will need this anyways IMO.

[evansde77]

evansde: patches reviewed and committed.

[DMWMBot]

rickw: Doesn't work. updateLFNsAndDatasets() gets called by all those setters, but it requires that all of those fields already be set.

Here's Bill & Ted's explanation:

Bill: Ted, while I agree that, in time, our band will be most triumphant. The truth is, Wyld Stallyns will never be a super band until we set the acquisition era. Ted: Yes, Bill. But, I do not believe we can set the acquisition era until we set the unmerged LFN base. Bill: Ted, it's pointless to set the unmerged LFN base before we set the processing version. Ted: Well, how can we set the processing version when we don't really even know the merged LFN base? Bill: That is why we NEED the acquisition era! Ted: And THAT is why we need the unmerged LFN base. Bill, Ted: EXCELLENT!

[evansde77]

evansde: /air guitar riff

[sfoulkes]

sfoulkes: This might not be the best API for setting up these attributes but I wanted to give you the option of changing them individually. They do all need to be set before the workflow is approved, but they can be set in whatever order you want. The unit test goes through setting them individually and verifies the nonsense datasets and lfns that get created when only the acquisition era or processing version is set: https://svnweb.cern.ch/trac/CMSDMWM/browser/WMCore/trunk/test/python/WMCore_t/WMSpec_t/WMWorkload_t.py#L360

[sfoulkes]

sfoulkes: Party on.

[DMWMBot]

rickw: I see. I was trying it against a most bogus older workload, which didn't have the defaults set.

[DMWMBot]

rickw: Which taskPath do I use to set the job and skim splitting parameters?

[sfoulkes]

sfoulkes: Added two more patches, 0004 reorganizes the ReReco and MonteCarlo workflow moving common code to the StdBase class. It also better integrates the ConfigCache and kills the code that would import the config and try to determine output modules.

As far as what TaskPath Rick needs to use the set the job and skim splitting parameters, the answer is all of them. I added a method in 0005 to list all of the tasks and job splitting parameters in a workflow.

[sfoulkes]

sfoulkes: And one more patch for a problem I just discovered.

[evansde77]

evansde: (In dd2c96c8c44da8b04555b94f7a1e2ca6aa14b1c2) Changes to workload StdBase and MC Workload StdSpec closes #260

combine final three patches from ticket 260

From: evansde77 evansde77@gmail.com

[DMWMBot]

rickw: Hi, Steve,

Could we add a little method in WMWorkloadHelper to loop over the internal tasks, and set their splitting parameters? The less the ReqMgr needs to know about the WMSpec internals, the better.