search

dmwm / WMCore

Core workflow management components for CMS.
Apache License 2.0
46 stars 107 forks source link

Paused jobs from replay using WMCore agent 1.5.2 with error SetupCMSSWPset #10804

Closed jhonatanamado closed 3 years ago

jhonatanamado commented 3 years ago

Impact of the bug Tier0 agent using 1.5.2

Describe the bug While testing WMCore agent 1.5.2, with the CMSSW 12_0_0, the replay shows paused jobs with an error within SetupCMSSWPset.py when trying to execute cmssw_wm_create_process.py.

How to reproduce it Replay using T0 release 3.0.0 based on 1.5.2 agent. Also logs, tarballs and sandbox to run an interactive job for the following CMSSW releases can be found as follows:

CMSSW 12_0_0 /afs/cern.ch/user/c/cmst0/public/PausedJobs/12_X_X/job_808

Expected behavior No errors from cmssw_wm_create_process.py

Additional context and error message Please take into account that Tier0 was able to deploy the replay using the 1.5.2 agent including the following PR #10801

germanfgv commented 3 years ago

This is the log that is produced when trying to run a T0 job with CMSSW_12_0_0. This error was produced using Pytho3 T0 Agent (WMCore 1.5.2):

2021-09-09 15:20:49,128:INFO:SetupCMSSWPset:Executing SetupCMSSWPSet...
2021-09-09 15:20:49,129:INFO:SetupCMSSWPset:Creating Scram environment with scram arch: slc7_amd64_gcc900 and CMSSW version: CMSSW_12_0_0
2021-09-09 15:20:52,154:INFO:SetupCMSSWPset:Setting up job scenario/process
2021-09-09 15:20:52,157:INFO:SetupCMSSWPset:ScramRun command args: cmssw_wm_create_process.py --output_pkl /srv/job/WMTaskSpace/cmsR
un1/PSet.pkl --funcname promptReco --funcargs /srv/job/WMTaskSpace/cmsRun1/process_funcArgs.json
2021-09-09 15:20:52,157:INFO:Scram:Creating a subprocess to run the PSet setup.
2021-09-09 15:20:52,157:INFO:Scram:Also recording SCRAM command-line related output.
2021-09-09 15:20:52,160:INFO:Scram:    Invoking command: cmssw_wm_create_process.py --output_pkl /srv/job/WMTaskSpace/cmsRun1/PSet.pkl --func
name promptReco --funcargs /srv/job/WMTaskSpace/cmsRun1/process_funcArgs.json
2021-09-09 15:20:53,237:INFO:Scram:Subprocess stdout was:
b\'Failed to load process from Scenario ppEra_Run2_2018 (<Configu
ration.DataProcessing.Impl.ppEra_Run2_2018.ppEra_Run2_2018 object at 0x2b98327a4790>).\
Traceback (most recent call last):\
  File "/cvmfs/cms.cern.ch/share/overrides/bin/cmssw_wm_create_process.py", l
ine 144, in <module>\
    main()\
  File "/cvmfs/cms.cern.ch/share/overrides/bin/cmssw_wm_create_process.py", line 135, in main\
    process=create_process(args, func_args)\
  File "/cvmfs/cms.cern.c
h/share/overrides/bin/cmssw_wm_create_process.py", line 97, in create_process\
    raise ex\
  File "/cvmfs/cms.cern.ch/share/overrides/bin/cmssw_wm_create_process.py", line 86, in create_process\

 arg_names=my_func.func_code.co_varnames[1:1+my_func.func_code.co_argcount]\
AttributeError: \\\'function\\\' object has no attribute \\\'func_code\\\'\
\'
2021-09-09 15:20:53,237:INFO:Scram:Subproces
s stderr was:
None
2021-09-09 15:20:53,237:ERROR:SetupCMSSWPset:Error running scram process. Error code: 5
2021-09-09 15:20:53,237:ERROR:SetupCMSSWPset:Error creating process for Config/DataProcessing
:
Traceback (most recent call last):
  File "/srv/job/WMCore.zip/WMCore/WMRuntime/Scripts/SetupCMSSWPset.py", line 695, in __call__
    self.createProcess(scenario, funcName, funcArgs)
  File "/srv/j
ob/WMCore.zip/WMCore/WMRuntime/Scripts/SetupCMSSWPset.py", line 156, in createProcess
    self.scramRun(cmd)
  File "/srv/job/WMCore.zip/WMCore/WMRuntime/Scripts/SetupCMSSWPset.py", line 119, in scramR
un
    raise RuntimeError(msg)
RuntimeError: Error running scram process. Error code: 5
'
 b'Traceback (most recent call last):
  File "/srv/job/WMCore.zip/WMCore/WMRuntime/ScriptInvoke.py", line 110, in <module>
  File "/srv/job/WMCore.zip/WMCore/WMRuntime/ScriptInvoke.py", line 81, in inv
oke
  File "/srv/job/WMCore.zip/WMCore/WMRuntime/Scripts/SetupCMSSWPset.py", line 698, in __call__
  File "/srv/job/WMCore.zip/WMCore/WMRuntime/Scripts/SetupCMSSWPset.py", line 695, in __call__
  File
 "/srv/job/WMCore.zip/WMCore/WMRuntime/Scripts/SetupCMSSWPset.py", line 156, in createProcess
  File "/srv/job/WMCore.zip/WMCore/WMRuntime/Scripts/SetupCMSSWPset.py", line 119, in scramRun
RuntimeError
: Error running scram process. Error code: 5
khurtado commented 3 years ago

This is an issue with the external script: cmssw_wm_create_process.py

Link to the external fix (no new Agent version needed, fixes are propagated via cvmfs to cmssw-wm-tools scripts): https://github.com/cms-sw/cmssw-wm-tools/commit/eb4568b379b688eaae5283140ed48b94252a9f6f

amaltaro commented 3 years ago

@khurtado @germanfgv if I'm not wrong, it was said over slack that this issue has now been fixed with the commit Kenyi points out above. If so, shall we close it? Thanks

khurtado commented 3 years ago

Yes, this can be closed (doing it now).