UnboundLocalError: local variable 'taskStatusCode' referenced before assignment

[todor-ivanov]

From times to times we see the following error in the crab server (REST interface) logs: [1]

[1] [07/Mar/2018:13:00:00] SERVER DATABASE ERROR 500/403 Execution error builtins.UnboundLocalError 3421356f61f664bdc117357af06889b7 [instance: prod] (local variable 'taskStatusCode' referenced before assignment); last statement: SELECT tm_taskname, panda_jobset_id, tm_task_status , tm_task_command, tm_user_role, tm_user_group, tm_task_failure, tm_split_args, panda_resubmitted_jobs, tm_save_logs, tm_username, tm_user_dn, tm_arguments, tm_input_dataset, tm_dbs_url, tm_task_warnings, tm_publication, tm_user_webdir, tm_asourl,
tm_asodb, tm_output_dataset, tm_collector, tm_schedd, tm_dry_run, clusterid, tm_start_time FROM tasks WHERE tm_taskname=:taskname; binds: (), {'taskname': '180301_201757:adow_crab_EphemeralHLTPhysics2'}; offset: None [07/Mar/2018:13:00:00] Traceback (most recent call last): [07/Mar/2018:13:00:00] File "/data/srv/beHG1803c/sw/slc7_amd64_gcc630/cms/crabserver/3.3.1711.rc4/lib/python2.7/site-packages/WMCore/REST/Server.py", line 1739, in dbapi_wrapper [07/Mar/2018:13:00:00] RESTSQL:VpUodabnEyPm disconnecting [07/Mar/2018:13:00:00] return handler(*xargs, xkwargs) [07/Mar/2018:13:00:00] File "/data/srv/beHG1803c/sw/slc7_amd64_gcc630/cms/crabserver/3.3.1711.rc4/lib/python2.7/site-packages/CRABInterface/RESTUserWorkflow.py", line 681, in get [07/Mar/2018:13:00:00] result = self.userworkflowmgr.status(workflow, verbose=verbose, userdn=userdn) [07/Mar/2018:13:00:00] File "/data/srv/beHG1803c/sw/slc7_amd64_gcc630/cms/crabserver/3.3.1711.rc4/lib/python2.7/site-packages/CRABInterface/DataUserWorkflow.py", line 173, in status [07/Mar/2018:13:00:00] return self.workflow.status(workflow, userdn, userproxy, verbose=verbose) [07/Mar/2018:13:00:00] File "/data/srv/beHG1803c/sw/slc7_amd64_gcc630/cms/crabserver/3.3.1711.rc4/lib/python2.7/site-packages/CRABInterface/Utils.py", line 186, in throttled_wrapped_function [07/Mar/2018:13:00:00] return fn(*args, *kw) [07/Mar/2018:13:00:00] File "/data/srv/beHG1803c/sw/slc7_amd64_gcc630/cms/crabserver/3.3.1711.rc4/lib/python2.7/site-packages/CRABInterface/Utils.py", line 144, in wrapped_func [07/Mar/2018:13:00:00] return func(args, kwargs) [07/Mar/2018:13:00:00] File "/data/srv/beHG1803c/sw/slc7_amd64_gcc630/cms/crabserver/3.3.1711.rc4/lib/python2.7/site-packages/CRABInterface/HTCondorDataWorkflow.py", line 677, in status [07/Mar/2018:13:00:00] RESTSQL:VpUodabnEyPm DISCONNECTED cms_analysis_reqmgr_r@cmsr timeout=300 inuse=4 idle=7 [07/Mar/2018:13:00:00] if taskStatusCode == 5: [07/Mar/2018:13:00:00] UnboundLocalError: local variable 'taskStatusCode' referenced before assignment

[belforte]

it is an old problem in HTCondorDataWorkflow.py definition of taskStatusCode is inside the "if useOldLogic" but the offending line 677 (and a few others) try to use it regardless of the OldLogic

but this code seems unchanced since years. Something else must have triggered the execution of a new path which hits the problematic lines.

Indeed PyCharm nicely points it out:

[belforte]

proper fix requires understanding of the logic of the code.

[belforte]

@todor-ivanov let me know if you want to pursue this or if I should add to my list

[todor-ivanov]

@belforte I would like to try