Description of close-out procedure

[vlimant]

here a GH issue to describe the close-out procedure, most of which is legacy and already known to everyone. Code is https://github.com/CMSCompOps/WmAgentScripts/blob/master/Unified/checkor.py

Here below the items in order of reading the code, picking up on all the things in "completed" reqmgr-status (with some option flags selecting some part of it)

• checking the list of things that need force-completing or by-passing, from afs files and mcm api
• getting the list of files which were invalidated by dynamo (for further information)

per workflow, the following things are don

• force completing if requested
• build requirement for custodial tape subscription based on campaign and tier rule
• force completing if over thresholds from campaign's rule
• collect information about ongoing ACDC
• collect information about required completion level by output dataset
• collect information about possibility of truncating the output dataset
• reduce the requirement of completion based on time in the system
• collect dataset parentage information, and apply cumulative threshold if configured as such
• check if all output are completed [A]
• verify whether the output can be set valid (but not closed-out)
• force complete if it can be tuncated
• verify that no lumisection is too big (according to campaign settings)
• collect information about tape custodial location and subscription
• collect information about number of files in phedex & dbs
• analyze the file count, compare with dynamo, invalidate in dbs some file if already not in phedex anymore and were globally deleted
• collect information about duplicated lumisections
• invalidate files that contain duplicates
• decide on a custodial tape destination based on parent, and campaign setting, and output size
• check for large fraction of invalidated file for tagging the wf
• fill summary information to the closedout-db in mongodb (towards making https://cms-unified.web.cern.ch/cms-unified//assistance.html)
• setting close-out if relevant making error reports otherwise and if relevant
• send notification to mcm based on the type of error

[amaltaro]

Thanks for providing these details, Jean-Roch. I think any mention of force-completing does not belong to this ticket and I'd like to decouple them. In this issue, we should start with work that is already in completed status, and then proceed with all those checks that you described above.

Do we provide the workflow summary / error report for every single workflow? Or only in cases the workflow didn't reach the target level of completion (let's say 90 or 95% for MC, etc).?

From your list, I'd say what we'd have to have in the close-out procedure is:

• build requirement for custodial tape subscription based on campaign and tier rule
  • similar to "collect information about tape custodial location and subscription". Or is there any difference?
  • similar to "decide on a custodial tape destination based on parent, and campaign setting, and output size". Can we join all these bullets into one clear description of how output subscriptions are handled?
• collect information about ongoing ACDC
  • (if there is any ACDC in acquired/running, then do nothing and try these checks in the next cycle(?)
• collect information about required completion level by output dataset
  • required completion by dataset? Where is it defined? or is it campaign-wise?
• collect dataset parentage information, and apply cumulative threshold if configured as such
  • what is the "cumulative threshold" you're talking here? please elaborate
• check if all output are completed [A]
  • do you mean are registered in both DBS and PhEDEx?
• verify whether the output can be set valid (but not closed-out)
  • don't we set output datasets to VALID when workflows get announced (or maybe when workflow moves to closed-out status)? If you're talking about setting output datasets to VALID while there is still some work/recovery on going, I'd handle that as an exception. Unless you have an algorithm that we could put in the system...
• verify that no lumisection is too big (according to campaign settings)
  • I don't get this point. This should be a check during assignment. If you're worried about Ops mistake, then I think we need to work on avoiding those mistakes and forget this check.
• collect information about number of files in phedex & dbs
  • what to do if they don't match?
• analyze the file count, compare with dynamo, invalidate in dbs some file if already not in phedex anymore and were globally deleted
  • why compare it with dynamo? which information is held by dynamo that we cannot have in PhEDEx and DBS?
• collect information about duplicated lumisections
  • invalidate files that contain duplicates. Then what do you do with those other healthy lumi sections that got invalidated in the same file? How you decide which file to invalidate? How do you decide whether recovery is needed after this invalidation?
• check for large fraction of invalidated file for tagging the wf
  • how this information is used? what do you mean by "tagging the wf"? Does it affect the final decision whether the workflow can be closed-out or not?
• fill summary information to the closedout-db in mongodb (towards making https://cms-unified.web.cern.ch/cms-unified//assistance.html)
  • this would add another dependency to the WMCore services. So we better handle it in a different way, either providing this summary via ReqMgr2/WMStats or in a different way.
• setting close-out if relevant making error reports otherwise and if relevant
• send notification to mcm based on the type of error
  • what is this notification? is it using a REST API and filling a McM request with some information?

I'm going to create another placeholder for auto force-completion with some of your bullets.

[ticoann]

Thank you very much Jean-Roch, Alan for creating this. I think it is good idea to keep separate tickets for (auto) force-complete and close out ticket. I will create one issue for moving from closed-out to announced as well. I know Jean-Roch is busy to write all the logic. It would be helpful brief explanation about conditions which make the each state transition and point the code as this ticket has. Alan and I can ask questions if something is not clear, hope we won't bother Jean-Roch too much with this. (But that is the penalty for being a only expert. 😄 FYI. Issue for announced transition #8921

[vlimant]

@amaltaro regarding force-complete : everything here is related to something in "completed". The force-completion would happen to on-going acdc, should the requirement be met. It is part of the current completed->close-out transition for the original workflow. Fine with me if you want to put it elsewhere though.

[vlimant]

many of the details above will have to come through reading the code to get the exact way. any description with words will likely fall short.

[vlimant]

you might find reading https://cms-unified.web.cern.ch/cms-unified/logs/checkor/last.log useful