ConfigCacheID output module validation

amaltaro commented 7 years ago

While debugging why a request was failing to get created in ReqMgr2, see json here [1], with this error message:

[24/May/2017:18:59:49] vocms0136.cern.ch 188.184.135.24 "GET /reqmgr2/data/request?name=etorassa_QCD-Fall11_R4-00024_T1_FR_CCIN2P3_MSS_v1_111229_145305 HTTP/1.1" 200 OK [data: 5972 in 1918 out 208826 us ] [auth: OK "/DC=ch/DC=cern/OU=com
puters/CN=pdmvserv/cmsdev04.cern.ch" "" ] [ref: "" "Python-urllib/2.6" ]
[24/May/2017:18:59:50]  Error: Traceback (most recent call last):
  File "/data/srv/beHG1705h/sw/slc6_amd64_gcc493/cms/reqmgr2/1.1.3_cmsweb.patch1/lib/python2.7/site-packages/WMCore/ReqMgr/Service/Request.py", line 189, in validate
    self._validateRequestBase(param, safe, validate_request_create_args)
  File "/data/srv/beHG1705h/sw/slc6_amd64_gcc493/cms/reqmgr2/1.1.3_cmsweb.patch1/lib/python2.7/site-packages/WMCore/ReqMgr/Service/Request.py", line 84, in _validateRequestBase
    workload, r_args = valFunc(args, self.config, self.reqmgr_db_service, param)
  File "/data/srv/beHG1705h/sw/slc6_amd64_gcc493/cms/reqmgr2/1.1.3_cmsweb.patch1/lib/python2.7/site-packages/WMCore/ReqMgr/Utils/Validation.py", line 144, in validate_request_create_args
    request_args)
  File "/data/srv/beHG1705h/sw/slc6_amd64_gcc493/cms/reqmgr2/1.1.3_cmsweb.patch1/lib/python2.7/site-packages/WMCore/WMSpec/StdSpecs/StdBase.py", line 847, in factoryWorkloadConstruction
    workload = self.__call__(workloadName=workloadName, arguments=arguments)
  File "/data/srv/beHG1705h/sw/slc6_amd64_gcc493/cms/reqmgr2/1.1.3_cmsweb.patch1/lib/python2.7/site-packages/WMCore/WMSpec/StdSpecs/TaskChain.py", line 271, in __call__
    self.setupTask(task, taskConf)
  File "/data/srv/beHG1705h/sw/slc6_amd64_gcc493/cms/reqmgr2/1.1.3_cmsweb.patch1/lib/python2.7/site-packages/WMCore/WMSpec/StdSpecs/TaskChain.py", line 151, in __call__
    self.func(self.obj, task, taskConf)
  File "/data/srv/beHG1705h/sw/slc6_amd64_gcc493/cms/reqmgr2/1.1.3_cmsweb.patch1/lib/python2.7/site-packages/WMCore/WMSpec/StdSpecs/TaskChain.py", line 412, in setupTask
    taskConf=taskConf)
  File "/data/srv/beHG1705h/sw/slc6_amd64_gcc493/cms/reqmgr2/1.1.3_cmsweb.patch1/lib/python2.7/site-packages/WMCore/WMSpec/StdSpecs/StdBase.py", line 420, in setupProcessingTask
    forceMerged=forceMerged, forceUnmerged=forceUnmerged, taskConf=taskConf)
  File "/data/srv/beHG1705h/sw/slc6_amd64_gcc493/cms/reqmgr2/1.1.3_cmsweb.patch1/lib/python2.7/site-packages/WMCore/WMSpec/StdSpecs/StdBase.py", line 511, in addOutputModule
    lfnBase(unmergedLFN)
  File "/data/srv/beHG1705h/sw/slc6_amd64_gcc493/cms/reqmgr2/1.1.3_cmsweb.patch1/lib/python2.7/site-packages/WMCore/Lexicon.py", line 413, in lfnBase
    return check(STORE_RESULTS_LFN, candidate)
  File "/data/srv/beHG1705h/sw/slc6_amd64_gcc493/cms/reqmgr2/1.1.3_cmsweb.patch1/lib/python2.7/site-packages/WMCore/Lexicon.py", line 466, in check
    "'%s' does not match regular expression %s" % (candidate, regexp)
AssertionError: '/store/unmerged/CMSSW_9_0_0/HLTPhysics/None/2017_05_24_18_56_HLTrefer_92X_dataRun2_HLT_v2-v1' does not match regular expression /store/results/([a-zA-Z0-9\-_]+)/([a-zA-Z0-9\-_]+)/([a-zA-Z0-9\.\-_]+)/([A-Z\-_]+)/([a-zA-Z0-9\-_]+)

I notice the Task1 configuration defines 4 outputModules but only 2 have the dataset parameter (thus the filterName and dataTier parameters). Since the hlt* ones don't have it, we validate the LFN with a None datatier.

My question is, is there any reason to accept such output module definition? Should we instead raise a WMSpec* exception and fail it with a clear message?

That code is very old, so probably there was a reason to make it how it was...

[1]

    "Group": "DATAOPS",
    "ProcessingString": "2017_05_24_19_24_HLTrefer_92X_dataRun2_HLT_v2",
    "Multicore": 1,
    "SizePerEvent": 2000,
    "Memory": 3000,
    "PrepID": "CMSSW_9_0_0__ALCARELVAL-2017_05_24_19_27_HLTPhysics_Run2017A_v1_RAW_refer",
    "DQMUploadUrl": "https://cmsweb.cern.ch/dqm/relval",
    "SubRequestType": "RelVal",
    "RequestType": "TaskChain",
    "TimePerEvent": 1.0,
    "AcquisitionEra": "CMSSW_9_0_0",
    "CMSSWVersion": "CMSSW_9_0_0",
    "EnableHarvesting": true,
    "RequestPriority": "900000",
    "DQMConfigCacheID": "b890061457596de35b30eb625fa84d31",
    "RequestString": "HLT_refer_RelVal_294929",
    "GlobalTag": "92X_dataRun2_HLT_v2",
    "DbsUrl": "https://cmsweb.cern.ch/dbs/prod/global/DBSReader",
    "ScramArch": "slc6_amd64_gcc530",
    "ConfigCacheUrl": "https://cmsweb.cern.ch/couchdb",
    "Campaign": "CMSSW_9_0_0__ALCARELVAL-2017_05_24_19_27",
    "Task1": {
        "InputDataset": "/HLTPhysics/Run2017A-v1/RAW",
        "KeepOutput": false,
        "Campaign": "CMSSW_9_0_0__ALCARELVAL-2017_05_24_19_27",
        "GlobalTag": "92X_dataRun2_HLT_v2",
        "LumiList": {"294929": [[1, 33]]},
        "SplittingAlgo": "LumiBased",
        "ProcessingString": "2017_05_24_19_24_HLTrefer_92X_dataRun2_HLT_v2",
        "ConfigCacheID": "b890061457596de35b30eb625fa60e4c",
        "LumisPerJob": 10,
        "TaskName": "Task1",
        "AcquisitionEra": "CMSSW_9_0_0"
    },
    "Task2": {
        "GlobalTag": "92X_dataRun2_PROMPT_ECalPulseShapes_w21_2017",
        "InputFromOutputModule": "FEVTDEBUGHLToutput",
        "ProcessingString": "2017_05_24_19_24_HLTrefer_92X_dataRun2_HLT_v2",
        "SplittingAlgo": "LumiBased",
        "InputTask": "Task1",
        "ConfigCacheID": "b890061457596de35b30eb625fa75bde",
        "LumisPerJob": 1,
        "TaskName": "Task2",
        "AcquisitionEra": "CMSSW_9_2_0_patch2",
        "Campaign": "CMSSW_9_0_0__ALCARELVAL-2017_05_24_19_27",
        "CMSSWVersion": "CMSSW_9_2_0_patch2"
    },
    "TaskChain": 2

amaltaro commented 7 years ago

I forgot to point where the pset configuration is validated: https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/Cache/WMConfigCache.py#L557

amaltaro commented 7 years ago

@ticoann since you're back, maybe you know this reason?

dmwm / WMCore

ConfigCacheID output module validation #7877