search

dmwm / CRABServer

15 stars 38 forks source link

#186: Test dev CRABClient using prod REST instance and CMSSW_7_6_7 CMSSW release #7022

Closed cmsdmwmbot closed 2 years ago

cmsdmwmbot commented 2 years ago

Tests started for following configuration:

Configuration:

Tests started:

cmsdmwmbot commented 2 years ago

Task submission for Client_Configuration_Validation successfully ended.

220131_101150:cmsbot_crab_transferOutputs
220131_101151:cmsbot_crab_transferLogs
220131_101152:cmsbot_crab_activity
220131_101153:cmsbot_crab_inputFiles
220131_101155:cmsbot_crab_disableAutomaticOutputCollection
220131_101156:cmsbot_crab_outputFiles
220131_101157:cmsbot_crab_allowUndistributedCMSSW
220131_101158:cmsbot_crab_maxMemoryMB
220131_101159:cmsbot_crab_maxJobRuntimeMin
220131_101200:cmsbot_crab_numCores
220131_101201:cmsbot_crab_scriptExe
220131_101202:cmsbot_crab_scriptArgs
220131_101205:cmsbot_crab_sendPythonFolder
220131_101209:cmsbot_crab_sendExternalFolder
220131_101210:cmsbot_crab_inputDBS
220131_101211:cmsbot_crab_useParent
220131_101212:cmsbot_crab_secondaryInputDataset
220131_101213:cmsbot_crab_lumiMaskFile
220131_101214:cmsbot_crab_runRange
220131_101215:cmsbot_crab_ignoreLocality
220131_101217:cmsbot_crab_userInputFiles
220131_101218:cmsbot_crab_whitelist
220131_101219:cmsbot_crab_blacklist
220131_101220:cmsbot_crab_ignoreGlobalBlacklist
220131_101221:cmsbot_crab_voRole
220131_101222:cmsbot_crab_voGroup
220131_101223:cmsbot_crab_scheddName
220131_101225:cmsbot_crab_collector
220131_101226:cmsbot_crab_extraJDL

Task submission for Client_Validation_Suite successfully ended.

220131_101149:cmsbot_crab_20220131_111147

Finished at: 2022-01-31 11:12:26 Find submission log here

cmsdmwmbot commented 2 years ago

Test: Client validation Result: SUCCEEDED Finished at: 2022-01-31 11:35:07 Test log: https://cmssdt.cern.ch/dmwm-jenkins/job/COPYCAT_CRABServer_ClientValidation/208/console All executed commands: https://cmssdt.cern.ch/dmwm-jenkins/job/COPYCAT_CRABServer_ClientValidation/208//artifact/client-validation.log/*view*/ Message: All commands were executed successfully.

cmsdmwmbot commented 2 years ago

Test: Client configuration validation Result: FULL-STATUS-UNKNOWN Attempt: 0 out of 4. Will run again. Finished at: 2022-01-31 11:42:16 Test log: https://cmssdt.cern.ch/dmwm-jenkins/job/COPYCAT_CRABServer_ClientConfigurationValidation/216/console

RETRY TESTS:
transferLogs-check.sh 220131_101151:cmsbot_crab_transferLogs - 2
maxMemoryMB-check.sh 220131_101158:cmsbot_crab_maxMemoryMB - 2
numCores-check.sh 220131_101200:cmsbot_crab_numCores - 2
scriptArgs-check.sh 220131_101202:cmsbot_crab_scriptArgs - 2
voGroup-check.sh 220131_101222:cmsbot_crab_voGroup - 2

SUCCESSFUL TESTS:
transferOutputs-check.sh 220131_101150:cmsbot_crab_transferOutputs - 0
activity-check.sh 220131_101152:cmsbot_crab_activity - 0
inputFiles-check.sh 220131_101153:cmsbot_crab_inputFiles - 0
disableAutomaticOutputCollection-check.sh 220131_101155:cmsbot_crab_disableAutomaticOutputCollection - 0
allowUndistributedCMSSW-check.sh 220131_101157:cmsbot_crab_allowUndistributedCMSSW - 0
maxJobRuntimeMin-check.sh 220131_101159:cmsbot_crab_maxJobRuntimeMin - 0
scriptExe-check.sh 220131_101201:cmsbot_crab_scriptExe - 0
sendPythonFolder-check.sh 220131_101205:cmsbot_crab_sendPythonFolder - 0
sendExternalFolder-check.sh 220131_101209:cmsbot_crab_sendExternalFolder - 0
inputDBS-check.sh 220131_101210:cmsbot_crab_inputDBS - 0
useParent-check.sh 220131_101211:cmsbot_crab_useParent - 0
blacklist-check.sh 220131_101219:cmsbot_crab_blacklist - 0
ignoreGlobalBlacklist-check.sh 220131_101220:cmsbot_crab_ignoreGlobalBlacklist - 0
scheddName-check.sh 220131_101223:cmsbot_crab_scheddName - 0
collector-check.sh 220131_101225:cmsbot_crab_collector - 0
extraJDL-check.sh 220131_101226:cmsbot_crab_extraJDL - 0

FAILED TESTS:
outputFiles-check.sh 220131_101156:cmsbot_crab_outputFiles - 1
secondaryInputDataset-check.sh 220131_101212:cmsbot_crab_secondaryInputDataset - 1
lumiMaskFile-check.sh 220131_101213:cmsbot_crab_lumiMaskFile - 1
runRange-check.sh 220131_101214:cmsbot_crab_runRange - 1
ignoreLocality-check.sh 220131_101215:cmsbot_crab_ignoreLocality - 1
userInputFiles-check.sh 220131_101217:cmsbot_crab_userInputFiles - 1
whitelist-check.sh 220131_101218:cmsbot_crab_whitelist - 1
voRole-check.sh 220131_101221:cmsbot_crab_voRole - 1
cmsdmwmbot commented 2 years ago

Test: Client configuration validation Result: FAILED Attempt: 1 out of 4. Test failed. Investigate manually Finished at: 2022-01-31 12:15:35 Test log: https://cmssdt.cern.ch/dmwm-jenkins/job/COPYCAT_CRABServer_ClientConfigurationValidation/223/console

RETRY TESTS:
 none

SUCCESSFUL TESTS:
transferOutputs-check.sh 220131_101150:cmsbot_crab_transferOutputs - 0
transferLogs-check.sh 220131_101151:cmsbot_crab_transferLogs - 0
activity-check.sh 220131_101152:cmsbot_crab_activity - 0
inputFiles-check.sh 220131_101153:cmsbot_crab_inputFiles - 0
disableAutomaticOutputCollection-check.sh 220131_101155:cmsbot_crab_disableAutomaticOutputCollection - 0
allowUndistributedCMSSW-check.sh 220131_101157:cmsbot_crab_allowUndistributedCMSSW - 0
maxMemoryMB-check.sh 220131_101158:cmsbot_crab_maxMemoryMB - 0
maxJobRuntimeMin-check.sh 220131_101159:cmsbot_crab_maxJobRuntimeMin - 0
numCores-check.sh 220131_101200:cmsbot_crab_numCores - 0
scriptExe-check.sh 220131_101201:cmsbot_crab_scriptExe - 0
scriptArgs-check.sh 220131_101202:cmsbot_crab_scriptArgs - 0
sendPythonFolder-check.sh 220131_101205:cmsbot_crab_sendPythonFolder - 0
sendExternalFolder-check.sh 220131_101209:cmsbot_crab_sendExternalFolder - 0
inputDBS-check.sh 220131_101210:cmsbot_crab_inputDBS - 0
useParent-check.sh 220131_101211:cmsbot_crab_useParent - 0
blacklist-check.sh 220131_101219:cmsbot_crab_blacklist - 0
ignoreGlobalBlacklist-check.sh 220131_101220:cmsbot_crab_ignoreGlobalBlacklist - 0
scheddName-check.sh 220131_101223:cmsbot_crab_scheddName - 0
collector-check.sh 220131_101225:cmsbot_crab_collector - 0
extraJDL-check.sh 220131_101226:cmsbot_crab_extraJDL - 0

FAILED TESTS:
outputFiles-check.sh 220131_101156:cmsbot_crab_outputFiles - 1
secondaryInputDataset-check.sh 220131_101212:cmsbot_crab_secondaryInputDataset - 1
lumiMaskFile-check.sh 220131_101213:cmsbot_crab_lumiMaskFile - 1
runRange-check.sh 220131_101214:cmsbot_crab_runRange - 1
ignoreLocality-check.sh 220131_101215:cmsbot_crab_ignoreLocality - 1
userInputFiles-check.sh 220131_101217:cmsbot_crab_userInputFiles - 1
whitelist-check.sh 220131_101218:cmsbot_crab_whitelist - 1
voRole-check.sh 220131_101221:cmsbot_crab_voRole - 1
voGroup-check.sh 220131_101222:cmsbot_crab_voGroup - 1
belforte commented 2 years ago

diagnotics of failed tests:

  1. outputFiles-check.sh 220131_101156:cmsbot_crab_outputFiles relies on crab getoutput i.e. gfal_copy which is not available inside singularity container
  2. secondaryInputDataset-check.sh 220131_101212:cmsbot_crab_secondaryInputDataset submission failed with error looking up DBS. I believe becasue the list of lumis in the secondary input /GenericTTbar/HC-CMSSW_9_2_6_91X_mcRun1_realistic_v2-v2/GEN-SIM-RAW and the primary input /GenericTTbar/HC-CMSSW_5_3_1_START53_V5-v1/GEN-SIM-RECO do not match. Better skip this test on ol CMSSW_7
  3. lumiMaskFile-check.sh 220131_101213:cmsbot_crab_lumiMaskFile current test uses a data lumiMask on a MC input file ! No way
  4. runRange-check.sh 220131_101214:cmsbot_crab_runRange same as above
  5. ignoreLocality-check.sh 220131_101215:cmsbot_crab_ignoreLocality test uses as input a file from CMSSW_8 which can't bre read in CMSSW_7
  6. userInputFiles-check.sh 220131_101217:cmsbot_crab_userInputFiles test uses as input a file from CMSSW_9 which can't be read in CMSSW_7
  7. whitelist-check.sh 220131_101218:cmsbot_crab_whitelist submission failed because input dataset is only at FNAL and test whitelist DESY
  8. voRole-check.sh 220131_101221:cmsbot_crab_voRole getlog command is called without --proxy which forces a proxy creation attempt inside crab client and this fails becasue myproxy-init is missing for CMSSW_7
  9. voGroup-check.sh 220131_101222:cmsbot_crab_voGroup same as voRole
belforte commented 2 years ago

action items from analysis of failures above:

belforte commented 2 years ago

all failures tracked to tests shortcoming to be fixed. Test successful