Add the CMS SE name to the stageOut step in the spec and modify the stage out runtime code

[DMWMBot]

To allow the async. stage-out of user outputs, it is needed to add the CMS SE name (something like asynchDestination attribute) to the stageOut step in the spec and modify the stage out runtime code to add the aynchDestination attribute to each output module in the FWJR.

[DMWMBot]

riahi: The user DN is also needed.

[drsm79]

metson: It needs to be the CMS name not the SE.

[sfoulkes]

sfoulkes: You'll need to do a couple things:

WMCore/WMSpec/WMStep.py

• In the WMStepHelper class, add methods to set the user DN and async destination
• Store them in self.data, eg: self.data.asyncDN, self.data.asyncDest

WMCore/WMSpec/StdSpecs/StdBase.py

• Modify the setupProcessing() method to take arguments for asyncDN and asyncDest
• Pass those parameters to the StageOut step using the helper methods that were created for this

WMCore/WMSpec/StdSpecs/Analysis.py

• Modify this to take asyncDN and asynDest as parameters and pass them to setupProcessing()

WMCore/WMSpec/Steps/Executor.py

• Modify the post() method to copy the asyncDN and asyncDest to the FWJR for each output module in the report

WMCore/FwkJobReport/Report.py

• Modify the to_json() method to make sure your new attributes are converted to JSON.

[drsm79]

metson: asyncSource should be included also - this should be the CMS name of the site the job ran at.

[DMWMBot]

riahi: Actually, the SE name of the site where the job ran is in FWJR and mapped therefore in JSM DB. The conversion from SE name to CMS name cannot be done in the WN (when the FWJR is built) since Phedex API in WMCore depend on httplib2 and it is not installed by default in WNs. So Maybe:

• it can just be inserted in JSM DB when FWJR is mapped there or;
• we can get the SE name from JSM DB and we convert it therefore in the async. module.

[evansde77]

evansde: Looking things up from a central service on the WN is basically a DDoS attack on that service with enough jobs.

The components should keep a cache of SE:CMS name somewhere and just do the conversion at the agent side when you pull it from the JSM. Doing this in the async module makes the most sense, since doing it in the JSM will do it for all jobs wether it is needed or not, which will introduce overhead.

[DMWMBot]

riahi: comment/question about this point:

WMCore/WMSpec/Steps/Executor.py

* Modify the post() method to copy the asyncDN and asyncDest to the FWJR for each output module in the report 

• I saw that the post method is already implemented for each step class...so the post method in each step class should be modified and not the post method in Executor class.
• Why don't we copy the asyncDN and asyncDest to the FWJR for each output in execute method rather than post method?

[evansde77]

evansde: Execute is for the execution of whatever you are doing, running binaries etc. Post is after execution logic, the main difference being that post can alter the flow of work in the job by returning the next step to execute.

In this case, you probably want to update the report with the async files after the job has run and you have made sure it was sucessful, which fits in with checking that things worked in the step specific post method

[DMWMBot]

riahi: Please review. Patches are enclosed to the ticket.