PromptSkimPoller - list index out of range

[samircury]

Hi,

We're almost sure that this problem was seen and solved before, but as it was raised yesterday and not much track of the discussion was kept as all was done in the chat, I'm creating this ticket to have it solved ASAP, as the hard part was done before, should be quick to fix.

{{{ sferedBlocks(): Running... 2012-03-09 11:30:13,897:INFO:PromptSkimPoller:pollForTransferedBlocks(): Found 1 blocks. 2012-03-09 11:30:13,897:INFO:PromptSkimPoller:pollForTransferedBlocks(): [{'STATUS': 'Active', 'PROCESSED_ID': 6, 'BLOCK_ID': 10, 'RUN_ID': 182915, 'BLOCKSIZE': 0, 'PRIMARY_ID': 2, 'BLOCK_NAME': '/HIHighPt/T0TEST_pstest_backfill1_cern_33_BUNNIES-PromptReco-v1/RECO#990d3988-687e-11e1-9a15-842b2b4671d8', 'MIGRATE_STATUS': 'NotMigrated', 'DATA_TIER': None, 'DATASET_ID': None, 'TIER_ID': 3, 'DATASET_PATH_ID': 13, 'FILECOUNT': 0}] 2012-03-09 11:30:13,897:INFO:PromptSkimPoller:pollForTransferedBlocks(): Skimmable: 10 2012-03-09 11:30:14,876:INFO:PromptSkimPoller:Error making workflows: list index out of range 2012-03-09 11:30:14,877:INFO:PromptSkimPoller:Traceback: Traceback (most recent call last): File "/data/srv/wmagent/v0.8.26pre6/sw/slc5_amd64_gcc461/cms/wmcore/0.8.26pre6/lib/python2.6/site-packages/WMComponent/PromptSkimScheduler/PromptSkimPoller.py", line 233, in pollForTransferedBlocks skimConfig, skimmableBlock) File "/data/srv/wmagent/v0.8.26pre6/sw/slc5_amd64_gcc461/cms/wmcore/0.8.26pre6/lib/python2.6/site-packages/WMComponent/PromptSkimScheduler/PromptSkimPoller.py", line 169, in createWorkloadsForBlock workload = self.promptSkimFactory(workloadName, wfParams) File "/data/srv/wmagent/v0.8.26pre6/sw/slc5_amd64_gcc461/cms/wmcore/0.8.26pre6/lib/python2.6/site-packages/WMCore/WMSpec/StdSpecs/PromptSkim.py", line 104, in call arguments["ProcConfigCacheID"] = configCache.getIDFromLabel(workloadName) File "/data/srv/wmagent/v0.8.26pre6/sw/slc5_amd64_gcc461/cms/wmcore/0.8.26pre6/lib/python2.6/site-packages/WMCore/Cache/WMConfigCache.py", line 471, in getIDFromLabel if results["rows"][0]["key"] == label: IndexError: list index out of range

}}}

Diego should follow up here

[DMWMBot]

dballest: There is another error showing up with MergeJobs, a problem in the stageOut.

{{{ Traceback: Traceback (most recent call last): File "/storage/local/data1/condor/execute/dir_28525/glide_G28933/execute/dir_31099/job/WMCore.zip/WMCore/Storage/StageOutMgr.py", line 297, in localStageOut impl(protocol, localPfn, pfn, options) File "/storage/local/data1/condor/execute/dir_28525/glide_G28933/execute/dir_31099/job/WMCore.zip/WMCore/Storage/StageOutImpl.py", line 180, in call raise ex StageOutError: StageOutError Message: Command exited non-zero ErrorCode : 60311 ModuleName : WMCore.Storage.StageOutError MethodName : init ErrorType : GeneralStageOutFailure ClassInstance : None FileName : /storage/local/data1/condor/execute/dir_28525/glide_G28933/execute/dir_31099/job/WMCore.zip/WMCore/Storage/StageOutError.py ClassName : None Command : #!/bin/sh . /opt/d-cache/dcap/bin/setenv-cmsprod.sh if [ ! -e "/pnfs/cms/WAX/11/store/data/T0TEST_pstest_backfill1_cern_34_BUNNIES/HIHighPt/RECO/PromptSkim-v1/0000" ]; then mkdir -p /pnfs/cms/WAX/11/store/data/T0TEST_pstest_backfill1_cern_34_BUNNIES/HIHighPt/RECO/PromptSkim-v1/0000 fi

LineNumber : 32
ErrorNr : 0
ExitCode : 1

Traceback: Traceback (most recent call last): File "/storage/local/data1/condor/execute/dir_28525/glide_G28933/execute/dir_31099/job/WMCore.zip/WMCore/Storage/StageOutImpl.py", line 169, in call self.createOutputDirectory(targetPFN) File "/storage/local/data1/condor/execute/dir_28525/glide_G28933/execute/dir_31099/job/WMCore.zip/WMCore/Storage/Backends/FNALImpl.py", line 120, in createOutputDirectory self.executeCommand(command) File "/storage/local/data1/condor/execute/dir_28525/glide_G28933/execute/dir_31099/job/WMCore.zip/WMCore/Storage/StageOutImpl.py", line 65, in executeCommand raise StageOutError(msg, Command = command, ExitCode = exitCode) StageOutError: StageOutError Message: Command exited non-zero ErrorCode : 60311 ModuleName : WMCore.Storage.StageOutError MethodName : init ErrorType : GeneralStageOutFailure ClassInstance : None FileName : /storage/local/data1/condor/execute/dir_28525/glide_G28933/execute/dir_31099/job/WMCore.zip/WMCore/Storage/StageOutError.py ClassName : None Command : #!/bin/sh . /opt/d-cache/dcap/bin/setenv-cmsprod.sh if [ ! -e "/pnfs/cms/WAX/11/store/data/T0TEST_pstest_backfill1_cern_34_BUNNIES/HIHighPt/RECO/PromptSkim-v1/0000" ]; then mkdir -p /pnfs/cms/WAX/11/store/data/T0TEST_pstest_backfill1_cern_34_BUNNIES/HIHighPt/RECO/PromptSkim-v1/0000 fi

}}}

There are more details in the elog:

https://cmslogbook.cern.ch/elog/Tier-0+processing/7591

[DMWMBot]

dballest: Configuration was set to write files to /store/data, and that's probably what caused the error because there are no tape families configured. I changed the MergedLFNBase to /store/backfill/1 and will test later with a new replay. I'll update here when it's done.

[sfoulkes]

sfoulkes: Everything seems to be working now, closing.