#343: Test prod CRABClient using test11 REST instance and CMSSW_7_6_7 CMSSW release

mapellidario / CRABServer

0 stars 0 forks source link

#343: Test prod CRABClient using test11 REST instance and CMSSW_7_6_7 CMSSW release #36

Closed cmsbuild closed 1 year ago

cmsbuild commented 1 year ago

Tests started for following configuration:

Configuration:

CRABClient_version: prod
- REST_Instance: test11
- CMSSW_release: CMSSW_7_6_7
- SCRAM_ARCH: slc6_amd64_gcc493

Tests started:

[ ] Client_Validation
[ ] Client_Configuration_Validation
[ ] Task_Submission_Status_Tracking

Started at: 2022-11-24 13:16:33 Link to the job

cmsbuild commented 1 year ago

Task submission for Task_Submission_Status_Tracking successfully ended.

221124_121746:cmsbot_crab_20221124_131744
221124_121748:cmsbot_crab_20221124_131746
221124_121750:cmsbot_crab_20221124_131748
221124_121752:cmsbot_crab_20221124_131750
221124_121754:cmsbot_crab_20221124_131752
221124_121812:cmsbot_crab_20221124_131754

Task submission for Client_Configuration_Validation successfully ended.

221124_121710:cmsbot_crab_transferOutputs
221124_121711:cmsbot_crab_transferLogs
221124_121712:cmsbot_crab_activity
221124_121713:cmsbot_crab_inputFiles
221124_121714:cmsbot_crab_disableAutomaticOutputCollection
221124_121715:cmsbot_crab_outputFiles
221124_121716:cmsbot_crab_allowUndistributedCMSSW
221124_121717:cmsbot_crab_maxMemoryMB
221124_121718:cmsbot_crab_maxJobRuntimeMin
221124_121719:cmsbot_crab_numCores
221124_121720:cmsbot_crab_scriptExe
221124_121721:cmsbot_crab_scriptArgs
221124_121723:cmsbot_crab_sendPythonFolder
221124_121727:cmsbot_crab_sendExternalFolder
221124_121728:cmsbot_crab_inputDBS
221124_121729:cmsbot_crab_useParent
221124_121730:cmsbot_crab_secondaryInputDataset
221124_121731:cmsbot_crab_lumiMaskFile
221124_121732:cmsbot_crab_outLFNDirBase
221124_121733:cmsbot_crab_runRange
221124_121734:cmsbot_crab_ignoreLocality
221124_121735:cmsbot_crab_userInputFiles
221124_121736:cmsbot_crab_whitelist
221124_121737:cmsbot_crab_blacklist
221124_121738:cmsbot_crab_ignoreGlobalBlacklist
221124_121739:cmsbot_crab_voRole
221124_121740:cmsbot_crab_voGroup
221124_121741:cmsbot_crab_scheddName
221124_121742:cmsbot_crab_collector
221124_121743:cmsbot_crab_extraJDL

Task submission for Client_Validation_Suite successfully ended.

221124_121709:cmsbot_crab_20221124_131707

Finished at: 2022-11-24 13:18:12 Find submission log here

cmsbuild commented 1 year ago

Test: Client validation Result: SUCCEEDED Finished at: 2022-11-24 13:39:22 Test log: https://cmssdt.cern.ch/dmwm-jenkins/job/CRABServer_ClientValidation/312/console All executed commands: https://cmssdt.cern.ch/dmwm-jenkins/job/CRABServer_ClientValidation/312//artifact/client-validation.log/*view*/ Message: All commands were executed successfully.

cmsbuild commented 1 year ago

Test: Client configuration validation Result: FULL-STATUS-UNKNOWN Attempt: 0 out of 4. Will run again. Finished at: 2022-11-24 13:40:20 Test log: https://cmssdt.cern.ch/dmwm-jenkins/job/CRABServer_ClientConfigurationValidation/512/console

RETRY TESTS:
transferLogs-check.sh 221124_121711:cmsbot_crab_transferLogs - 2
activity-check.sh 221124_121712:cmsbot_crab_activity - 2
maxMemoryMB-check.sh 221124_121717:cmsbot_crab_maxMemoryMB - 2
maxJobRuntimeMin-check.sh 221124_121718:cmsbot_crab_maxJobRuntimeMin - 2
numCores-check.sh 221124_121719:cmsbot_crab_numCores - 2
scriptExe-check.sh 221124_121720:cmsbot_crab_scriptExe - 2
scriptArgs-check.sh 221124_121721:cmsbot_crab_scriptArgs - 2
blacklist-check.sh 221124_121737:cmsbot_crab_blacklist - 2
ignoreGlobalBlacklist-check.sh 221124_121738:cmsbot_crab_ignoreGlobalBlacklist - 2
voRole-check.sh 221124_121739:cmsbot_crab_voRole - 2
voGroup-check.sh 221124_121740:cmsbot_crab_voGroup - 2

SUCCESSFUL TESTS:
transferOutputs-check.sh 221124_121710:cmsbot_crab_transferOutputs - 0
inputFiles-check.sh 221124_121713:cmsbot_crab_inputFiles - 0
disableAutomaticOutputCollection-check.sh 221124_121714:cmsbot_crab_disableAutomaticOutputCollection - 0
outputFiles-check.sh 221124_121715:cmsbot_crab_outputFiles - 0
allowUndistributedCMSSW-check.sh 221124_121716:cmsbot_crab_allowUndistributedCMSSW - 0
sendPythonFolder-check.sh 221124_121723:cmsbot_crab_sendPythonFolder - 0
sendExternalFolder-check.sh 221124_121727:cmsbot_crab_sendExternalFolder - 0
inputDBS-check.sh 221124_121728:cmsbot_crab_inputDBS - 0
secondaryInputDataset-check.sh 221124_121730:cmsbot_crab_secondaryInputDataset - 0
lumiMaskFile-check.sh 221124_121731:cmsbot_crab_lumiMaskFile - 0
outLFNDirBase-check.sh 221124_121732:cmsbot_crab_outLFNDirBase - 0
runRange-check.sh 221124_121733:cmsbot_crab_runRange - 0
ignoreLocality-check.sh 221124_121734:cmsbot_crab_ignoreLocality - 0
userInputFiles-check.sh 221124_121735:cmsbot_crab_userInputFiles - 0
whitelist-check.sh 221124_121736:cmsbot_crab_whitelist - 0
scheddName-check.sh 221124_121741:cmsbot_crab_scheddName - 0
collector-check.sh 221124_121742:cmsbot_crab_collector - 0
extraJDL-check.sh 221124_121743:cmsbot_crab_extraJDL - 0

FAILED TESTS:
useParent-check.sh 221124_121729:cmsbot_crab_useParent - 1

cmsbuild commented 1 year ago

Test: Client configuration validation Result: FULL-STATUS-UNKNOWN Attempt: 1 out of 4. Will run again. Finished at: 2022-11-24 14:02:27 Test log: https://cmssdt.cern.ch/dmwm-jenkins/job/CRABServer_ClientConfigurationValidation/514/console

RETRY TESTS:
blacklist-check.sh 221124_121737:cmsbot_crab_blacklist - 2

SUCCESSFUL TESTS:
transferOutputs-check.sh 221124_121710:cmsbot_crab_transferOutputs - 0
transferLogs-check.sh 221124_121711:cmsbot_crab_transferLogs - 0
activity-check.sh 221124_121712:cmsbot_crab_activity - 0
inputFiles-check.sh 221124_121713:cmsbot_crab_inputFiles - 0
disableAutomaticOutputCollection-check.sh 221124_121714:cmsbot_crab_disableAutomaticOutputCollection - 0
outputFiles-check.sh 221124_121715:cmsbot_crab_outputFiles - 0
allowUndistributedCMSSW-check.sh 221124_121716:cmsbot_crab_allowUndistributedCMSSW - 0
maxMemoryMB-check.sh 221124_121717:cmsbot_crab_maxMemoryMB - 0
maxJobRuntimeMin-check.sh 221124_121718:cmsbot_crab_maxJobRuntimeMin - 0
numCores-check.sh 221124_121719:cmsbot_crab_numCores - 0
scriptExe-check.sh 221124_121720:cmsbot_crab_scriptExe - 0
scriptArgs-check.sh 221124_121721:cmsbot_crab_scriptArgs - 0
sendPythonFolder-check.sh 221124_121723:cmsbot_crab_sendPythonFolder - 0
sendExternalFolder-check.sh 221124_121727:cmsbot_crab_sendExternalFolder - 0
inputDBS-check.sh 221124_121728:cmsbot_crab_inputDBS - 0
secondaryInputDataset-check.sh 221124_121730:cmsbot_crab_secondaryInputDataset - 0
lumiMaskFile-check.sh 221124_121731:cmsbot_crab_lumiMaskFile - 0
outLFNDirBase-check.sh 221124_121732:cmsbot_crab_outLFNDirBase - 0
runRange-check.sh 221124_121733:cmsbot_crab_runRange - 0
ignoreLocality-check.sh 221124_121734:cmsbot_crab_ignoreLocality - 0
userInputFiles-check.sh 221124_121735:cmsbot_crab_userInputFiles - 0
whitelist-check.sh 221124_121736:cmsbot_crab_whitelist - 0
ignoreGlobalBlacklist-check.sh 221124_121738:cmsbot_crab_ignoreGlobalBlacklist - 0
voRole-check.sh 221124_121739:cmsbot_crab_voRole - 0
voGroup-check.sh 221124_121740:cmsbot_crab_voGroup - 0
scheddName-check.sh 221124_121741:cmsbot_crab_scheddName - 0
collector-check.sh 221124_121742:cmsbot_crab_collector - 0
extraJDL-check.sh 221124_121743:cmsbot_crab_extraJDL - 0

FAILED TESTS:
useParent-check.sh 221124_121729:cmsbot_crab_useParent - 1

cmsbuild commented 1 year ago

Test: Task Submission Status Tracking Result: FULL-STATUS-UNKNOWN Attempt: 0 out of 4. Will run again. Finished at: 2022-11-24 14:20:55 Test log: https://cmssdt.cern.ch/dmwm-jenkins/job/CRABServer_CheckTestResults/378/console

{'TN': '221124_121746:cmsbot_crab_20221124_131744', 'dbStatus': u'SUBMITTED', 'testResult': 'TestRunning', 'jobsPerStatus': {'finished': 994, 'transferring': 6}, 'combinedStatus': 'SUBMITTED'}
{'TN': '221124_121748:cmsbot_crab_20221124_131746', 'dbStatus': u'SUBMITTED', 'testResult': 'TestRunning', 'jobsPerStatus': {'finished': 5, 'running': 1}, 'combinedStatus': 'SUBMITTED'}
{'TN': '221124_121750:cmsbot_crab_20221124_131748', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 2}, 'combinedStatus': 'COMPLETED'}
{'TN': '221124_121752:cmsbot_crab_20221124_131750', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 10}, 'combinedStatus': 'COMPLETED'}
{'TN': '221124_121754:cmsbot_crab_20221124_131752', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 100}, 'combinedStatus': 'COMPLETED'}
{'TN': '221124_121812:cmsbot_crab_20221124_131754', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 10}, 'combinedStatus': 'COMPLETED'}

cmsbuild commented 1 year ago

Test: Client configuration validation Result: FAILED Attempt: 2 out of 4. Test failed. Investigate manually Finished at: 2022-11-24 14:23:39 Test log: https://cmssdt.cern.ch/dmwm-jenkins/job/CRABServer_ClientConfigurationValidation/517/console

RETRY TESTS:
 none

SUCCESSFUL TESTS:
transferOutputs-check.sh 221124_121710:cmsbot_crab_transferOutputs - 0
transferLogs-check.sh 221124_121711:cmsbot_crab_transferLogs - 0
activity-check.sh 221124_121712:cmsbot_crab_activity - 0
inputFiles-check.sh 221124_121713:cmsbot_crab_inputFiles - 0
disableAutomaticOutputCollection-check.sh 221124_121714:cmsbot_crab_disableAutomaticOutputCollection - 0
outputFiles-check.sh 221124_121715:cmsbot_crab_outputFiles - 0
allowUndistributedCMSSW-check.sh 221124_121716:cmsbot_crab_allowUndistributedCMSSW - 0
maxMemoryMB-check.sh 221124_121717:cmsbot_crab_maxMemoryMB - 0
maxJobRuntimeMin-check.sh 221124_121718:cmsbot_crab_maxJobRuntimeMin - 0
numCores-check.sh 221124_121719:cmsbot_crab_numCores - 0
scriptExe-check.sh 221124_121720:cmsbot_crab_scriptExe - 0
scriptArgs-check.sh 221124_121721:cmsbot_crab_scriptArgs - 0
sendPythonFolder-check.sh 221124_121723:cmsbot_crab_sendPythonFolder - 0
sendExternalFolder-check.sh 221124_121727:cmsbot_crab_sendExternalFolder - 0
inputDBS-check.sh 221124_121728:cmsbot_crab_inputDBS - 0
secondaryInputDataset-check.sh 221124_121730:cmsbot_crab_secondaryInputDataset - 0
lumiMaskFile-check.sh 221124_121731:cmsbot_crab_lumiMaskFile - 0
outLFNDirBase-check.sh 221124_121732:cmsbot_crab_outLFNDirBase - 0
runRange-check.sh 221124_121733:cmsbot_crab_runRange - 0
ignoreLocality-check.sh 221124_121734:cmsbot_crab_ignoreLocality - 0
userInputFiles-check.sh 221124_121735:cmsbot_crab_userInputFiles - 0
whitelist-check.sh 221124_121736:cmsbot_crab_whitelist - 0
blacklist-check.sh 221124_121737:cmsbot_crab_blacklist - 0
ignoreGlobalBlacklist-check.sh 221124_121738:cmsbot_crab_ignoreGlobalBlacklist - 0
voRole-check.sh 221124_121739:cmsbot_crab_voRole - 0
voGroup-check.sh 221124_121740:cmsbot_crab_voGroup - 0
scheddName-check.sh 221124_121741:cmsbot_crab_scheddName - 0
collector-check.sh 221124_121742:cmsbot_crab_collector - 0
extraJDL-check.sh 221124_121743:cmsbot_crab_extraJDL - 0

FAILED TESTS:
useParent-check.sh 221124_121729:cmsbot_crab_useParent - 1

cmsbuild commented 1 year ago

Test: Task Submission Status Tracking Result: FULL-STATUS-UNKNOWN Attempt: 1 out of 4. Will run again. Finished at: 2022-11-24 15:23:52 Test log: https://cmssdt.cern.ch/dmwm-jenkins/job/CRABServer_CheckTestResults/382/console

{'TN': '221124_121746:cmsbot_crab_20221124_131744', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 1000}, 'combinedStatus': 'COMPLETED'}
{'TN': '221124_121748:cmsbot_crab_20221124_131746', 'dbStatus': u'SUBMITTED', 'testResult': 'TestRunning', 'jobsPerStatus': {'finished': 5, 'transferring': 1}, 'combinedStatus': 'SUBMITTED'}
{'TN': '221124_121750:cmsbot_crab_20221124_131748', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 2}, 'combinedStatus': 'COMPLETED'}
{'TN': '221124_121752:cmsbot_crab_20221124_131750', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 10}, 'combinedStatus': 'COMPLETED'}
{'TN': '221124_121754:cmsbot_crab_20221124_131752', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 100}, 'combinedStatus': 'COMPLETED'}
{'TN': '221124_121812:cmsbot_crab_20221124_131754', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 10}, 'combinedStatus': 'COMPLETED'}

cmsbuild commented 1 year ago

Test: Task Submission Status Tracking Result: FAILED Attempt: 2 out of 4. Test failed. Investigate manually Finished at: 2022-11-24 16:27:45 Test log: https://cmssdt.cern.ch/dmwm-jenkins/job/CRABServer_CheckTestResults/383/console

{'TN': '221124_121746:cmsbot_crab_20221124_131744', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 1000}, 'combinedStatus': 'COMPLETED'}
{'TN': '221124_121748:cmsbot_crab_20221124_131746', 'dbStatus': u'SUBMITTED', 'testResult': 'TestFailed', 'jobsPerStatus': {'failed': 1, 'finished': 5, 'transferring': 1}, 'combinedStatus': 'SUBMITTED'}
{'TN': '221124_121750:cmsbot_crab_20221124_131748', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 2}, 'combinedStatus': 'COMPLETED'}
{'TN': '221124_121752:cmsbot_crab_20221124_131750', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 10}, 'combinedStatus': 'COMPLETED'}
{'TN': '221124_121754:cmsbot_crab_20221124_131752', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 100}, 'combinedStatus': 'COMPLETED'}
{'TN': '221124_121812:cmsbot_crab_20221124_131754', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 10}, 'combinedStatus': 'COMPLETED'}

mapellidario commented 1 year ago

Some tasks failed due to file read error, resubmitting them

[ ] 221124_121729:cmsbot_crab_useParent
[x] 221124_121748:cmsbot_crab_20221124_131746, crab status is ok

mapellidario commented 1 year ago

@belforte quick question. I am testing my PR https://github.com/dmwm/CRABServer/pull/7441 here.

Is it expected that when the parent dataset is only on tape, the job fails because the redirector can not find the input file? Have I been unlucky enough that the file was on disk when jenkins submitted the job, but disappeared before the job could start?

we are using as input dataset a GEN-SIM-RECO dataset:

CMSSW_release=CMSSW_7_6_7;SCRAM_ARCH=slc6_amd64_gcc493;inputDataset=/GenericTTbar/HC-CMSSW_5_3_1_START53_V5-v1/GEN-SIM-RECO;master=no;

The job log is here, which complains about not finding a GEN-SIM-RAW file

== CMSSW:       [g] Last URL tried: root://cms-xrd-global.cern.ch:1094//store/mc/HC/GenericTTbar/GEN-SIM-RAW/CMSSW_5_3_1_START53_V5-v1/0010/2C3473DF-BBAD-E111-B401-0025901D626C.root?tried=+1213cmsxrootd2.fnal.gov1213xrootd.unl.edu,

which is in the parent GEN-SIM-RAW dataset

> dasgoclient --query="dataset file=/store/mc/HC/GenericTTbar/GEN-SIM-RAW/CMSSW_5_3_1_START53_V5-v1/0010/2C3473DF-BBAD-E111-B401-0025901D626C.root"
/GenericTTbar/HC-CMSSW_5_3_1_START53_V5-v1/GEN-SIM-RAW

That GEN-SIM-RAW dataset is the parent of out input dataset GEN-SIM-RECO and it is only on tape

> dasgoclient --query="parent dataset=/GenericTTbar/HC-CMSSW_5_3_1_START53_V5-v1/GEN-SIM-RECO"
/GenericTTbar/HC-CMSSW_5_3_1_START53_V5-v1/GEN-SIM-RAW
> dasgoclient --query="site dataset=/GenericTTbar/HC-CMSSW_5_3_1_START53_V5-v1/GEN-SIM-RAW"
T1_US_FNAL_Tape

belforte commented 1 year ago

Is it expected that when the parent dataset is only on tape, the job fails because the redirector can not find the input file?

yes. There's no xroot access to tapes

belforte commented 1 year ago

Have I been unlucky enough that the file was on disk when jenkins submitted the job, but disappeared before the job could start?

No. Parent is on tape only all of the time. submission only checks location of input dataset, it is up to the person who asks for parents to make sure that parents are on disk.

belforte commented 1 year ago

please see: https://github.com/dmwm/CRABServer/issues/7456