search

dmwm / WMCore

Core workflow management components for CMS.
Apache License 2.0
46 stars 107 forks source link

MSOutputProducer crashing with UnboundLocalError #9726

Closed amaltaro closed 4 years ago

amaltaro commented 4 years ago

Impact of the bug ReqMgr2MS (MSOutputProducer)

Describe the bug It looks like MSOutputProducer fails to create the campaign/output map for very old workflows, which in general is Okay because we do not have very old/test workflows in the production system (if we do, we can get them archived). However, we should change the code such that it doesn't crash if it finds a very old workflow without the ChainParentageMap attribute.

How to reproduce it The ChainParentageMap was inserted into the requests around 2018, so any workflows before that will be able to reproduce this error.

Expected behavior For such workflows, we can use the top level 'Campaign' attribute and map all of the output datasets to it. It won't be completely correct, but that information will not be used anyways for data insertion.

Additional context and error message Fresh traceback from the logs:

2020-06-06 04:34:43,698:INFO:MSOutput: Running the msOutputProducer ...
2020-06-06 04:34:43,698:INFO:MSOutputStreamer: Could not open bufferFile: None. Exception coercing to Unicode: need string or buffer, NoneType foundFalling back to dictionary buffer.
2020-06-06 04:34:43,713:ERROR:MSOutput: ERR: Unable to create MSOutputTemplate for document {u'AcquisitionEra': {u'DigiFullPU_2017PU': u'CMSSW_9_0_2',
...
 u'RequestName': u'amaltaro_RVCMSSW_9_0_2TTbarLepton_13_PU25ns__HS_2017_resub_Converted_170602_105551_3888',
...
 u'_id': u'amaltaro_RVCMSSW_9_0_2TTbarLepton_13_PU25ns__HS_2017_resub_Converted_170602_105551_3888'}ERR: 'ChainParentageMap'
2020-06-06 04:34:43,713:ERROR:MSOutput: MSOutputProducer PipelineNonRelVal General Error from pipeline. Err: local variable 'msOutDoc' referenced before assignment. Giving up Now.
2020-06-06 04:34:43,713:ERROR:MSOutput: local variable 'msOutDoc' referenced before assignment
Traceback (most recent call last):
  File "/data/srv/beHG2006h/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.2.5.pre5/lib/python2.7/site-packages/WMCore/MicroService/Unified/MSOutput.py", line 588, in msOutputProducer
    pipeLine.run(request)
  File "/data/srv/beHG2006h/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.2.5.pre5/lib/python2.7/site-packages/Utils/Pipeline.py", line 139, in run
    return reduce(lambda obj, functor: functor(obj), self.funcLine, obj)
  File "/data/srv/beHG2006h/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.2.5.pre5/lib/python2.7/site-packages/Utils/Pipeline.py", line 139, in <lambda>
    return reduce(lambda obj, functor: functor(obj), self.funcLine, obj)
  File "/data/srv/beHG2006h/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.2.5.pre5/lib/python2.7/site-packages/Utils/Pipeline.py", line 71, in __call__
    return self.run(obj)
  File "/data/srv/beHG2006h/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.2.5.pre5/lib/python2.7/site-packages/Utils/Pipeline.py", line 74, in run
    return self.func(obj, *self.args, **self.kwargs)
  File "/data/srv/beHG2006h/sw/slc7_amd64_gcc630/cms/reqmgr2ms/0.2.5.pre5/lib/python2.7/site-packages/WMCore/MicroService/Unified/MSOutput.py", line 625, in docTransformer
    return msOutDoc
UnboundLocalError: local variable 'msOutDoc' referenced before assignment
2020-06-06 04:34:43,714:INFO:MSOutput: MSOutputProducer:140182741645056@vocms0731.cern.ch: Total 14 requests processed from the streamer. 
2020-06-06 04:34:44,042:INFO:MSManager: Total outputProducer execution time: 18 secs
amaltaro commented 4 years ago

@todor-ivanov I think you were already aware of this bug, but I could not find any GH issues made for it, so here it is.