search

dmwm / WMCore

Core workflow management components for CMS.
Apache License 2.0
46 stars 107 forks source link

Extend WMSpec produced output support to TFileService and arbitrary files #2697

Closed spigad closed 12 years ago

spigad commented 12 years ago

Currently we only support OutputModule, For the analysis we need to being able to manage both TFIleService outputs and arbitrary files from the user.

[from #2683]

ericvaandering commented 12 years ago

ewv: Anyone have opinions on where in the /store/user name space we should put these files?

For reference, the logs are being stored in /pnfs/cms/WAX/11/store/user/ewv/CRAB-Logs/[workflow-name]/

ericvaandering commented 12 years ago

ewv: Mostly my notes for where to go with this so I don't lose track over the weekend.

CRABClient is modified to only tell CS about the extra files, not the EDM files in output modules.

Analysis.py puts notice of the lfnBase and file names into the workflow using the Template/CMSSW

Executor CMSSW loads the workflow (already) and puts the correct entries into the Report.pkl Report.py needs to be changed as well, probably. This step determines the correct destination LFN for the file which includes the jobcounter (number 1..N).

Finally, Executor/StageOut is modified to pick up the extra output files and copy them to /store/temp/user.

From there this becomes an AsyncStageout task.

ericvaandering commented 12 years ago

ewv: Matt, would you mind taking a look at this patch and telling me if I am doing something stupid somewhere? It does work and do what I want.

DMWMBot commented 12 years ago

mnorman: Work? On a Friday?

I did look at it and for the most part it looks fine, as long as you're sure it does what you want. One thing I worry about though is that in the CMSSW step you hardcode:

{{{self.report.getAnalysisFilesFromStep('cmsRun1')}}}

How does this perform in situations where people have chained multiple CMSSW steps together?

Also, I should look at it again before the final commit. I've made some changes to StdSpecs.Analysis which haven't been checked in yet, and I should try and make sure things don't conflict.

ericvaandering commented 12 years ago

ewv: Also modify to_json in Report.py to spit out analysis files as well to get it into couch.

ericvaandering commented 12 years ago

ewv: Final version of patch is on #2330.

sfoulkes commented 12 years ago

sfoulkes: (In 0b70ec8234860572388b72e3cd5697341268ae5e) Support for producing arbitary files. Fixes #2330, #2697, #2783.

From: Eric Vaandering ewv@fnal.gov Signed-off-by: Steve Foulkes sfoulkes@fnal.gov