improve transition to "done" status

[vlimant]

referring to https://cms-pdmv-prod.web.cern.ch/mcm/requests?prepid=BPH-Run3Summer22EEDRPremix-00154 and https://gitlab.cern.ch/cms-ppd/dataset-management/dm-coordination/-/issues/4 the transition is stuck because a request was force completed and the output dataset is in limbo (0 events, and unknown in DBS) ; an exception should be created for this

[vlimant]

linking https://gitlab.cern.ch/cms-ppd/dataset-management/dm-coordination/-/issues/4 where the concern first arose

[lmoureaux]

The request (log) was force-closed before any events could be produced:

completor | Mon Feb 26 03:14:02 2024
    mcm is asking for cmsunified_task_BPH-Run3Summer22EEGS-00130__v1_T_240226_005606_7185 to be force complete

closor | Mon Feb 26 04:52:17 2024
    /K0LToMuMu_K0LFilter_TuneCP5_13p6TeV_pythia8-evtgen/Run3Summer22EEDRPremix-124X_mcRun3_2022_realistic_postEE_v1-v3/AODSIM 0/9349 = 0.00%
    /K0LToMuMu_K0LFilter_TuneCP5_13p6TeV_pythia8-evtgen/Run3Summer22EEMiniAODv4-130X_mcRun3_2022_realistic_postEE_v6-v3/MINIAODSIM 0/9349 = 0.00%
    /K0LToMuMu_K0LFilter_TuneCP5_13p6TeV_pythia8-evtgen/Run3Summer22EENanoAODv12-130X_mcRun3_2022_realistic_postEE_v6-v3/NANOAODSIM 0/9349 = 0.00%
    workflow outputs are announced

The current status is the logical outcome of this and, by CompOps books, the task was successful so they won't attempt to run it again.

It is unclear why the request was force-completed and the McM history hasn't recorded a trace of this.

By this analysis:

• There is a flaw in the workflow state machine, which should refuse to force-complete workflows before any events are produced (only the abort operation should be available). This leads to workflows in normal-archived without events in DBS.
• McM tries to outguess the CompOps status by checking the dataset status in DBS (collect_status_and_completed_events).
• Requests that are done by CompOps but not valid in DBS get stuck in an intermediate state. I do not think that McM should mark them as completed nor send them back to new. Ideally we'd have a separate status for CompOps-related failures. I'm very wary of touching the McM data model, for the code is full of undocumented assumptions, so I do not recommend implementing this in current McM.