#146: Test prod CRABClient using preprod REST instance and CMSSW_12_0_1 CMSSW release

cmsbuild commented 2 years ago

Tests started for following configuration:

Configuration:

CRABClient_version: prod
- REST_Instance: preprod
- CMSSW_release: CMSSW_12_0_1
- SCRAM_ARCH: slc7_amd64_gcc900

Tests started:

[ ] Client_Validation
[ ] Client_Configuration_Validation
[ ] Task_Submission_Status_Tracking

Started at: 2021-12-20 10:17:33 Link to the job

cmsbuild commented 2 years ago

Task submission for Task_Submission_Status_Tracking successfully ended.

211220_092120:cmsbot_crab_20211220_102119
211220_092124:cmsbot_crab_20211220_102121
211220_092127:cmsbot_crab_20211220_102125
211220_092129:cmsbot_crab_20211220_102127
211220_092132:cmsbot_crab_20211220_102130
211220_092139:cmsbot_crab_20211220_102133

Task submission for Client_Configuration_Validation successfully ended.

211220_091930:cmsbot_crab_transferOutputs
211220_091933:cmsbot_crab_transferLogs
211220_091936:cmsbot_crab_activity
211220_091938:cmsbot_crab_inputFiles
211220_091941:cmsbot_crab_disableAutomaticOutputCollection
211220_091944:cmsbot_crab_outputFiles
211220_091947:cmsbot_crab_allowUndistributedCMSSW
211220_091949:cmsbot_crab_maxMemoryMB
211220_091952:cmsbot_crab_maxJobRuntimeMin
211220_091955:cmsbot_crab_numCores
211220_091957:cmsbot_crab_scriptExe
211220_092000:cmsbot_crab_scriptArgs
211220_092003:cmsbot_crab_sendPythonFolder
211220_092013:cmsbot_crab_sendExternalFolder
211220_092015:cmsbot_crab_inputDBS
211220_092018:cmsbot_crab_useParent
211220_092021:cmsbot_crab_secondaryInputDataset
211220_092024:cmsbot_crab_lumiMaskFile
211220_092026:cmsbot_crab_lumiMaskUrl
211220_092050:cmsbot_crab_outLFNDirBase
211220_092053:cmsbot_crab_runRange
211220_092055:cmsbot_crab_ignoreLocality
211220_092058:cmsbot_crab_userInputFiles
211220_092101:cmsbot_crab_whitelist
211220_092104:cmsbot_crab_blacklist
211220_092107:cmsbot_crab_ignoreGlobalBlacklist
211220_092109:cmsbot_crab_voRole
211220_092112:cmsbot_crab_scheddName
211220_092115:cmsbot_crab_collector
211220_092117:cmsbot_crab_extraJDL

Task submission for Client_Validation_Suite successfully ended.

211220_091927:cmsbot_crab_20211220_101921

Finished at: 2021-12-20 10:21:39 Find submission log here

cmsbuild commented 2 years ago

Test: Client validation Result: SUCCEEDED Finished at: 2021-12-20 10:45:54 Test log: https://cmssdt.cern.ch/dmwm-jenkins/job/COPYCAT_CRABServer_ClientValidation/175/console All executed commands: https://cmssdt.cern.ch/dmwm-jenkins/job/COPYCAT_CRABServer_ClientValidation/175//artifact/client-validation.log/*view*/ Message: All commands were executed successfully.

cmsbuild commented 2 years ago

Test: Client configuration validation Result: FULL-STATUS-UNKNOWN Attempt: 0 out of 4. Will run again. Finished at: 2021-12-20 10:47:23 Test log: https://cmssdt.cern.ch/dmwm-jenkins/job/COPYCAT_CRABServer_ClientConfigurationValidation/108/console

RETRY TESTS:
scriptExe-check.sh 211220_091957:cmsbot_crab_scriptExe - 2
useParent-check.sh 211220_092018:cmsbot_crab_useParent - 2
lumiMaskUrl-check.sh 211220_092026:cmsbot_crab_lumiMaskUrl - 2
runRange-check.sh 211220_092053:cmsbot_crab_runRange - 2
whitelist-check.sh 211220_092101:cmsbot_crab_whitelist - 2
ignoreGlobalBlacklist-check.sh 211220_092107:cmsbot_crab_ignoreGlobalBlacklist - 2

SUCCESSFUL TESTS:
transferOutputs-check.sh 211220_091930:cmsbot_crab_transferOutputs - 0
transferLogs-check.sh 211220_091933:cmsbot_crab_transferLogs - 0
activity-check.sh 211220_091936:cmsbot_crab_activity - 0
inputFiles-check.sh 211220_091938:cmsbot_crab_inputFiles - 0
disableAutomaticOutputCollection-check.sh 211220_091941:cmsbot_crab_disableAutomaticOutputCollection - 0
outputFiles-check.sh 211220_091944:cmsbot_crab_outputFiles - 0
allowUndistributedCMSSW-check.sh 211220_091947:cmsbot_crab_allowUndistributedCMSSW - 0
maxMemoryMB-check.sh 211220_091949:cmsbot_crab_maxMemoryMB - 0
maxJobRuntimeMin-check.sh 211220_091952:cmsbot_crab_maxJobRuntimeMin - 0
numCores-check.sh 211220_091955:cmsbot_crab_numCores - 0
scriptArgs-check.sh 211220_092000:cmsbot_crab_scriptArgs - 0
sendPythonFolder-check.sh 211220_092003:cmsbot_crab_sendPythonFolder - 0
sendExternalFolder-check.sh 211220_092013:cmsbot_crab_sendExternalFolder - 0
inputDBS-check.sh 211220_092015:cmsbot_crab_inputDBS - 0
secondaryInputDataset-check.sh 211220_092021:cmsbot_crab_secondaryInputDataset - 0
lumiMaskFile-check.sh 211220_092024:cmsbot_crab_lumiMaskFile - 0
outLFNDirBase-check.sh 211220_092050:cmsbot_crab_outLFNDirBase - 0
ignoreLocality-check.sh 211220_092055:cmsbot_crab_ignoreLocality - 0
userInputFiles-check.sh 211220_092058:cmsbot_crab_userInputFiles - 0
blacklist-check.sh 211220_092104:cmsbot_crab_blacklist - 0
voRole-check.sh 211220_092109:cmsbot_crab_voRole - 0
scheddName-check.sh 211220_092112:cmsbot_crab_scheddName - 0
extraJDL-check.sh 211220_092117:cmsbot_crab_extraJDL - 0

FAILED TESTS:
collector-check.sh 211220_092115:cmsbot_crab_collector - 1

cmsbuild commented 2 years ago

Test: Client configuration validation Result: FULL-STATUS-UNKNOWN Attempt: 1 out of 4. Will run again. Finished at: 2021-12-20 11:13:00 Test log: https://cmssdt.cern.ch/dmwm-jenkins/job/COPYCAT_CRABServer_ClientConfigurationValidation/109/console

RETRY TESTS:
scriptExe-check.sh 211220_091957:cmsbot_crab_scriptExe - 2
useParent-check.sh 211220_092018:cmsbot_crab_useParent - 2
ignoreGlobalBlacklist-check.sh 211220_092107:cmsbot_crab_ignoreGlobalBlacklist - 2

SUCCESSFUL TESTS:
transferOutputs-check.sh 211220_091930:cmsbot_crab_transferOutputs - 0
transferLogs-check.sh 211220_091933:cmsbot_crab_transferLogs - 0
activity-check.sh 211220_091936:cmsbot_crab_activity - 0
inputFiles-check.sh 211220_091938:cmsbot_crab_inputFiles - 0
disableAutomaticOutputCollection-check.sh 211220_091941:cmsbot_crab_disableAutomaticOutputCollection - 0
outputFiles-check.sh 211220_091944:cmsbot_crab_outputFiles - 0
allowUndistributedCMSSW-check.sh 211220_091947:cmsbot_crab_allowUndistributedCMSSW - 0
maxMemoryMB-check.sh 211220_091949:cmsbot_crab_maxMemoryMB - 0
maxJobRuntimeMin-check.sh 211220_091952:cmsbot_crab_maxJobRuntimeMin - 0
numCores-check.sh 211220_091955:cmsbot_crab_numCores - 0
scriptArgs-check.sh 211220_092000:cmsbot_crab_scriptArgs - 0
sendPythonFolder-check.sh 211220_092003:cmsbot_crab_sendPythonFolder - 0
sendExternalFolder-check.sh 211220_092013:cmsbot_crab_sendExternalFolder - 0
inputDBS-check.sh 211220_092015:cmsbot_crab_inputDBS - 0
secondaryInputDataset-check.sh 211220_092021:cmsbot_crab_secondaryInputDataset - 0
lumiMaskFile-check.sh 211220_092024:cmsbot_crab_lumiMaskFile - 0
lumiMaskUrl-check.sh 211220_092026:cmsbot_crab_lumiMaskUrl - 0
outLFNDirBase-check.sh 211220_092050:cmsbot_crab_outLFNDirBase - 0
runRange-check.sh 211220_092053:cmsbot_crab_runRange - 0
ignoreLocality-check.sh 211220_092055:cmsbot_crab_ignoreLocality - 0
userInputFiles-check.sh 211220_092058:cmsbot_crab_userInputFiles - 0
whitelist-check.sh 211220_092101:cmsbot_crab_whitelist - 0
blacklist-check.sh 211220_092104:cmsbot_crab_blacklist - 0
voRole-check.sh 211220_092109:cmsbot_crab_voRole - 0
scheddName-check.sh 211220_092112:cmsbot_crab_scheddName - 0
extraJDL-check.sh 211220_092117:cmsbot_crab_extraJDL - 0

FAILED TESTS:
collector-check.sh 211220_092115:cmsbot_crab_collector - 1

cmsbuild commented 2 years ago

Test: Task Submission Status Tracking Result: FULL-STATUS-UNKNOWN Attempt: 0 out of 4. Will run again. Finished at: 2021-12-20 11:24:25 Test log: https://cmssdt.cern.ch/dmwm-jenkins/job/COPYCAT_CRABServer_CheckTestResults/84/console

{'TN': '211220_092120:cmsbot_crab_20211220_102119', 'dbStatus': u'SUBMITTED', 'testResult': 'TestFailed', 'jobsPerStatus': {'failed': 2, 'finished': 996, 'running': 2}, 'combinedStatus': 'SUBMITTED'}
{'TN': '211220_092124:cmsbot_crab_20211220_102121', 'dbStatus': u'SUBMITTED', 'testResult': 'TestRunning', 'jobsPerStatus': {'finished': 5, 'running': 2}, 'combinedStatus': 'SUBMITTED'}
{'TN': '211220_092127:cmsbot_crab_20211220_102125', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 1}, 'combinedStatus': 'COMPLETED'}
{'TN': '211220_092129:cmsbot_crab_20211220_102127', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 10}, 'combinedStatus': 'COMPLETED'}
{'TN': '211220_092132:cmsbot_crab_20211220_102130', 'dbStatus': u'SUBMITTED', 'testResult': 'TestRunning', 'jobsPerStatus': {'finished': 94, 'transferring': 6}, 'combinedStatus': 'SUBMITTED'}
{'TN': '211220_092139:cmsbot_crab_20211220_102133', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 10}, 'combinedStatus': 'COMPLETED'}

cmsbuild commented 2 years ago

Test: Client configuration validation Result: FULL-STATUS-UNKNOWN Attempt: 2 out of 4. Will run again. Finished at: 2021-12-20 11:38:24 Test log: https://cmssdt.cern.ch/dmwm-jenkins/job/COPYCAT_CRABServer_ClientConfigurationValidation/110/console

RETRY TESTS:
scriptExe-check.sh 211220_091957:cmsbot_crab_scriptExe - 2
useParent-check.sh 211220_092018:cmsbot_crab_useParent - 2
ignoreGlobalBlacklist-check.sh 211220_092107:cmsbot_crab_ignoreGlobalBlacklist - 2

SUCCESSFUL TESTS:
transferOutputs-check.sh 211220_091930:cmsbot_crab_transferOutputs - 0
transferLogs-check.sh 211220_091933:cmsbot_crab_transferLogs - 0
activity-check.sh 211220_091936:cmsbot_crab_activity - 0
inputFiles-check.sh 211220_091938:cmsbot_crab_inputFiles - 0
disableAutomaticOutputCollection-check.sh 211220_091941:cmsbot_crab_disableAutomaticOutputCollection - 0
outputFiles-check.sh 211220_091944:cmsbot_crab_outputFiles - 0
allowUndistributedCMSSW-check.sh 211220_091947:cmsbot_crab_allowUndistributedCMSSW - 0
maxMemoryMB-check.sh 211220_091949:cmsbot_crab_maxMemoryMB - 0
maxJobRuntimeMin-check.sh 211220_091952:cmsbot_crab_maxJobRuntimeMin - 0
numCores-check.sh 211220_091955:cmsbot_crab_numCores - 0
scriptArgs-check.sh 211220_092000:cmsbot_crab_scriptArgs - 0
sendPythonFolder-check.sh 211220_092003:cmsbot_crab_sendPythonFolder - 0
sendExternalFolder-check.sh 211220_092013:cmsbot_crab_sendExternalFolder - 0
inputDBS-check.sh 211220_092015:cmsbot_crab_inputDBS - 0
secondaryInputDataset-check.sh 211220_092021:cmsbot_crab_secondaryInputDataset - 0
lumiMaskFile-check.sh 211220_092024:cmsbot_crab_lumiMaskFile - 0
lumiMaskUrl-check.sh 211220_092026:cmsbot_crab_lumiMaskUrl - 0
outLFNDirBase-check.sh 211220_092050:cmsbot_crab_outLFNDirBase - 0
runRange-check.sh 211220_092053:cmsbot_crab_runRange - 0
ignoreLocality-check.sh 211220_092055:cmsbot_crab_ignoreLocality - 0
userInputFiles-check.sh 211220_092058:cmsbot_crab_userInputFiles - 0
whitelist-check.sh 211220_092101:cmsbot_crab_whitelist - 0
blacklist-check.sh 211220_092104:cmsbot_crab_blacklist - 0
voRole-check.sh 211220_092109:cmsbot_crab_voRole - 0
scheddName-check.sh 211220_092112:cmsbot_crab_scheddName - 0
extraJDL-check.sh 211220_092117:cmsbot_crab_extraJDL - 0

FAILED TESTS:
collector-check.sh 211220_092115:cmsbot_crab_collector - 1

cmsbuild commented 2 years ago

Test: Client configuration validation Result: FULL-STATUS-UNKNOWN Attempt: 3 out of 4. Will run again. Finished at: 2021-12-20 12:03:49 Test log: https://cmssdt.cern.ch/dmwm-jenkins/job/COPYCAT_CRABServer_ClientConfigurationValidation/112/console

RETRY TESTS:
scriptExe-check.sh 211220_091957:cmsbot_crab_scriptExe - 2
useParent-check.sh 211220_092018:cmsbot_crab_useParent - 2
ignoreGlobalBlacklist-check.sh 211220_092107:cmsbot_crab_ignoreGlobalBlacklist - 2

SUCCESSFUL TESTS:
transferOutputs-check.sh 211220_091930:cmsbot_crab_transferOutputs - 0
transferLogs-check.sh 211220_091933:cmsbot_crab_transferLogs - 0
activity-check.sh 211220_091936:cmsbot_crab_activity - 0
inputFiles-check.sh 211220_091938:cmsbot_crab_inputFiles - 0
disableAutomaticOutputCollection-check.sh 211220_091941:cmsbot_crab_disableAutomaticOutputCollection - 0
outputFiles-check.sh 211220_091944:cmsbot_crab_outputFiles - 0
allowUndistributedCMSSW-check.sh 211220_091947:cmsbot_crab_allowUndistributedCMSSW - 0
maxMemoryMB-check.sh 211220_091949:cmsbot_crab_maxMemoryMB - 0
maxJobRuntimeMin-check.sh 211220_091952:cmsbot_crab_maxJobRuntimeMin - 0
numCores-check.sh 211220_091955:cmsbot_crab_numCores - 0
scriptArgs-check.sh 211220_092000:cmsbot_crab_scriptArgs - 0
sendPythonFolder-check.sh 211220_092003:cmsbot_crab_sendPythonFolder - 0
sendExternalFolder-check.sh 211220_092013:cmsbot_crab_sendExternalFolder - 0
inputDBS-check.sh 211220_092015:cmsbot_crab_inputDBS - 0
secondaryInputDataset-check.sh 211220_092021:cmsbot_crab_secondaryInputDataset - 0
lumiMaskFile-check.sh 211220_092024:cmsbot_crab_lumiMaskFile - 0
lumiMaskUrl-check.sh 211220_092026:cmsbot_crab_lumiMaskUrl - 0
outLFNDirBase-check.sh 211220_092050:cmsbot_crab_outLFNDirBase - 0
runRange-check.sh 211220_092053:cmsbot_crab_runRange - 0
ignoreLocality-check.sh 211220_092055:cmsbot_crab_ignoreLocality - 0
userInputFiles-check.sh 211220_092058:cmsbot_crab_userInputFiles - 0
whitelist-check.sh 211220_092101:cmsbot_crab_whitelist - 0
blacklist-check.sh 211220_092104:cmsbot_crab_blacklist - 0
voRole-check.sh 211220_092109:cmsbot_crab_voRole - 0
scheddName-check.sh 211220_092112:cmsbot_crab_scheddName - 0
extraJDL-check.sh 211220_092117:cmsbot_crab_extraJDL - 0

FAILED TESTS:
collector-check.sh 211220_092115:cmsbot_crab_collector - 1

cmsbuild commented 2 years ago

Test: Task Submission Status Tracking Result: FULL-STATUS-UNKNOWN Attempt: 1 out of 4. Will run again. Finished at: 2021-12-20 12:27:33 Test log: https://cmssdt.cern.ch/dmwm-jenkins/job/COPYCAT_CRABServer_CheckTestResults/86/console

{'TN': '211220_092120:cmsbot_crab_20211220_102119', 'dbStatus': u'SUBMITTED', 'testResult': 'TestFailed', 'jobsPerStatus': {'failed': 4, 'finished': 996}, 'combinedStatus': 'FAILED'}
{'TN': '211220_092124:cmsbot_crab_20211220_102121', 'dbStatus': u'SUBMITTED', 'testResult': 'TestRunning', 'jobsPerStatus': {'finished': 6, 'running': 1}, 'combinedStatus': 'SUBMITTED'}
{'TN': '211220_092127:cmsbot_crab_20211220_102125', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 1}, 'combinedStatus': 'COMPLETED'}
{'TN': '211220_092129:cmsbot_crab_20211220_102127', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 10}, 'combinedStatus': 'COMPLETED'}
{'TN': '211220_092132:cmsbot_crab_20211220_102130', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 100}, 'combinedStatus': 'COMPLETED'}
{'TN': '211220_092139:cmsbot_crab_20211220_102133', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 10}, 'combinedStatus': 'COMPLETED'}

cmsbuild commented 2 years ago

Test: Client configuration validation Result: FAILED Attempt: 4 out of 4. Test failed. Investigate manually Finished at: 2021-12-20 12:29:13 Test log: https://cmssdt.cern.ch/dmwm-jenkins/job/COPYCAT_CRABServer_ClientConfigurationValidation/113/console

RETRY TESTS:
 none

SUCCESSFUL TESTS:
transferOutputs-check.sh 211220_091930:cmsbot_crab_transferOutputs - 0
transferLogs-check.sh 211220_091933:cmsbot_crab_transferLogs - 0
activity-check.sh 211220_091936:cmsbot_crab_activity - 0
inputFiles-check.sh 211220_091938:cmsbot_crab_inputFiles - 0
disableAutomaticOutputCollection-check.sh 211220_091941:cmsbot_crab_disableAutomaticOutputCollection - 0
outputFiles-check.sh 211220_091944:cmsbot_crab_outputFiles - 0
allowUndistributedCMSSW-check.sh 211220_091947:cmsbot_crab_allowUndistributedCMSSW - 0
maxMemoryMB-check.sh 211220_091949:cmsbot_crab_maxMemoryMB - 0
maxJobRuntimeMin-check.sh 211220_091952:cmsbot_crab_maxJobRuntimeMin - 0
numCores-check.sh 211220_091955:cmsbot_crab_numCores - 0
scriptArgs-check.sh 211220_092000:cmsbot_crab_scriptArgs - 0
sendPythonFolder-check.sh 211220_092003:cmsbot_crab_sendPythonFolder - 0
sendExternalFolder-check.sh 211220_092013:cmsbot_crab_sendExternalFolder - 0
inputDBS-check.sh 211220_092015:cmsbot_crab_inputDBS - 0
secondaryInputDataset-check.sh 211220_092021:cmsbot_crab_secondaryInputDataset - 0
lumiMaskFile-check.sh 211220_092024:cmsbot_crab_lumiMaskFile - 0
lumiMaskUrl-check.sh 211220_092026:cmsbot_crab_lumiMaskUrl - 0
outLFNDirBase-check.sh 211220_092050:cmsbot_crab_outLFNDirBase - 0
runRange-check.sh 211220_092053:cmsbot_crab_runRange - 0
ignoreLocality-check.sh 211220_092055:cmsbot_crab_ignoreLocality - 0
userInputFiles-check.sh 211220_092058:cmsbot_crab_userInputFiles - 0
whitelist-check.sh 211220_092101:cmsbot_crab_whitelist - 0
blacklist-check.sh 211220_092104:cmsbot_crab_blacklist - 0
voRole-check.sh 211220_092109:cmsbot_crab_voRole - 0
scheddName-check.sh 211220_092112:cmsbot_crab_scheddName - 0
extraJDL-check.sh 211220_092117:cmsbot_crab_extraJDL - 0

FAILED TESTS:
scriptExe-check.sh 211220_091957:cmsbot_crab_scriptExe - 1
useParent-check.sh 211220_092018:cmsbot_crab_useParent - 1
ignoreGlobalBlacklist-check.sh 211220_092107:cmsbot_crab_ignoreGlobalBlacklist - 1
collector-check.sh 211220_092115:cmsbot_crab_collector - 1

ddaina / variousStuff

#146: Test prod CRABClient using preprod REST instance and CMSSW_12_0_1 CMSSW release #101