search

novicecpp / CRABServer

0 stars 0 forks source link

#365: Test GH CRABClient using preprod REST instance and CMSSW_12_6_0_pre4 CMSSW release #8

Closed cmsbuild closed 1 year ago

cmsbuild commented 1 year ago

Tests started for following configuration:

Configuration:

Tests started:

Started at: 2022-12-08 12:05:33 Link to the job

cmsbuild commented 1 year ago

Task submission for Task_Submission_Status_Tracking successfully ended.

221208_110754:cmsbot_crab_20221208_120754
221208_110756:cmsbot_crab_20221208_120755
221208_110757:cmsbot_crab_20221208_120756
221208_110758:cmsbot_crab_20221208_120758
221208_110800:cmsbot_crab_20221208_120759
221208_110827:cmsbot_crab_20221208_120801

Task submission for Client_Configuration_Validation successfully ended.

221208_110701:cmsbot_crab_transferOutputs
221208_110703:cmsbot_crab_transferLogs
221208_110704:cmsbot_crab_activity
221208_110706:cmsbot_crab_inputFiles
221208_110707:cmsbot_crab_disableAutomaticOutputCollection
221208_110709:cmsbot_crab_outputFiles
221208_110710:cmsbot_crab_allowUndistributedCMSSW
221208_110711:cmsbot_crab_maxMemoryMB
221208_110713:cmsbot_crab_maxJobRuntimeMin
221208_110714:cmsbot_crab_numCores
221208_110716:cmsbot_crab_scriptExe
221208_110717:cmsbot_crab_scriptArgs
221208_110719:cmsbot_crab_sendPythonFolder
221208_110727:cmsbot_crab_sendExternalFolder
221208_110728:cmsbot_crab_inputDBS
221208_110730:cmsbot_crab_useParent
221208_110731:cmsbot_crab_secondaryInputDataset
221208_110733:cmsbot_crab_lumiMaskFile
221208_110734:cmsbot_crab_lumiMaskUrl
221208_110736:cmsbot_crab_outLFNDirBase
221208_110738:cmsbot_crab_runRange
221208_110739:cmsbot_crab_ignoreLocality
221208_110741:cmsbot_crab_userInputFiles
221208_110742:cmsbot_crab_whitelist
221208_110744:cmsbot_crab_blacklist
221208_110745:cmsbot_crab_ignoreGlobalBlacklist
221208_110747:cmsbot_crab_voRole
221208_110748:cmsbot_crab_voGroup
221208_110749:cmsbot_crab_scheddName
221208_110751:cmsbot_crab_collector
221208_110752:cmsbot_crab_extraJDL

Task submission for Client_Validation_Suite successfully ended.

221208_110700:cmsbot_crab_20221208_120658

Finished at: 2022-12-08 12:08:28 Find submission log here

cmsbuild commented 1 year ago

Test: Client validation Result: FAILED Finished at: 2022-12-08 12:31:48 Test log: https://cmssdt.cern.ch/dmwm-jenkins/job/CRABServer_ClientValidation/329/console All executed commands: https://cmssdt.cern.ch/dmwm-jenkins/job/CRABServer_ClientValidation/329//artifact/client-validation.log/*view*/ Message: Some CRABClient commands failed. Find failed commands below:

TEST_COMMAND: crab status  --long --proxy=/tmp/x509up_u501  --dir=crab_20221208_120658  
TEST_RESULT: FAILED                                                                                 
TEST_MESSAGE:
CRAB project directory:     /build/cmsbld/jenkins/workspace/CRABServer_ClientValidation/CRABServer/test/container/testingScripts/crab_20221208_120658
Task name:          221208_110700:cmsbot_crab_20221208_120658
Grid scheduler - Task Worker:   crab3@vocms059.cern.ch - crab-preprod-tw01
Status on the CRAB server:  SUBMITTED
Task URL to use for HELP:   https://cmsweb-testbed.cern.ch/crabserver/ui/task/221208_110700%3Acmsbot_crab_20221208_120658
Dashboard monitoring URL:   https://monit-grafana.cern.ch/d/cmsTMDetail/cms-task-monitoring-task-view?orgId=11&var-user=cmsbot&var-task=221208_110700%3Acmsbot_crab_20221208_120658&from=1670494020000&to=now
Status on the scheduler:    FAILED

Jobs status:                    failed               10.0% ( 1/10)
                finished             90.0% ( 9/10)
Fatal error trying to connect to https://cmsweb-testbed.cern.ch:8443/crabserver/preprod/workflow?subresource=publicationstatus&workflow=221208_110700%3Acmsbot_crab_20221208_120658&username=cmsbot using subresource=publicationstatus&workflow=221208_110700%3Acmsbot_crab_20221208_120658&username=cmsbot.
The server answered with an error.
Server answered with: Database unavailable
Error Id: 6c3ab2db2ba41f9b880bb0da9eff9ae2
cmsbuild commented 1 year ago

Test: Client configuration validation Result: FAILED Attempt: 0 out of 4. Test failed. Investigate manually Finished at: 2022-12-08 12:41:19 Test log: https://cmssdt.cern.ch/dmwm-jenkins/job/CRABServer_ClientConfigurationValidation/557/console

RETRY TESTS:
 none

SUCCESSFUL TESTS:
 none

FAILED TESTS:
transferOutputs-check.sh 221208_110701:cmsbot_crab_transferOutputs - 1
transferLogs-check.sh 221208_110703:cmsbot_crab_transferLogs - 1
activity-check.sh 221208_110704:cmsbot_crab_activity - 1
inputFiles-check.sh 221208_110706:cmsbot_crab_inputFiles - 1
disableAutomaticOutputCollection-check.sh 221208_110707:cmsbot_crab_disableAutomaticOutputCollection - 1
outputFiles-check.sh 221208_110709:cmsbot_crab_outputFiles - 1
allowUndistributedCMSSW-check.sh 221208_110710:cmsbot_crab_allowUndistributedCMSSW - 1
maxMemoryMB-check.sh 221208_110711:cmsbot_crab_maxMemoryMB - 1
maxJobRuntimeMin-check.sh 221208_110713:cmsbot_crab_maxJobRuntimeMin - 1
numCores-check.sh 221208_110714:cmsbot_crab_numCores - 1
scriptExe-check.sh 221208_110716:cmsbot_crab_scriptExe - 1
scriptArgs-check.sh 221208_110717:cmsbot_crab_scriptArgs - 1
sendPythonFolder-check.sh 221208_110719:cmsbot_crab_sendPythonFolder - 1
sendExternalFolder-check.sh 221208_110727:cmsbot_crab_sendExternalFolder - 1
inputDBS-check.sh 221208_110728:cmsbot_crab_inputDBS - 1
useParent-check.sh 221208_110730:cmsbot_crab_useParent - 1
secondaryInputDataset-check.sh 221208_110731:cmsbot_crab_secondaryInputDataset - 1
lumiMaskFile-check.sh 221208_110733:cmsbot_crab_lumiMaskFile - 1
lumiMaskUrl-check.sh 221208_110734:cmsbot_crab_lumiMaskUrl - 1
outLFNDirBase-check.sh 221208_110736:cmsbot_crab_outLFNDirBase - 1
runRange-check.sh 221208_110738:cmsbot_crab_runRange - 1
ignoreLocality-check.sh 221208_110739:cmsbot_crab_ignoreLocality - 1
userInputFiles-check.sh 221208_110741:cmsbot_crab_userInputFiles - 1
whitelist-check.sh 221208_110742:cmsbot_crab_whitelist - 1
blacklist-check.sh 221208_110744:cmsbot_crab_blacklist - 1
ignoreGlobalBlacklist-check.sh 221208_110745:cmsbot_crab_ignoreGlobalBlacklist - 1
voRole-check.sh 221208_110747:cmsbot_crab_voRole - 1
voGroup-check.sh 221208_110748:cmsbot_crab_voGroup - 1
scheddName-check.sh 221208_110749:cmsbot_crab_scheddName - 1
collector-check.sh 221208_110751:cmsbot_crab_collector - 1
extraJDL-check.sh 221208_110752:cmsbot_crab_extraJDL - 1
cmsbuild commented 1 year ago

Test: Task Submission Status Tracking Result: FULL-STATUS-UNKNOWN Attempt: 0 out of 4. Will run again. Finished at: 2022-12-08 13:11:01 Test log: https://cmssdt.cern.ch/dmwm-jenkins/job/CRABServer_CheckTestResults/406/console

{'TN': '221208_110754:cmsbot_crab_20221208_120754', 'dbStatus': u'SUBMITTED', 'testResult': 'TestFailed', 'jobsPerStatus': {'failed': 54, 'finished': 918, 'transferring': 27, 'running': 1}, 'combinedStatus': 'SUBMITTED'}
{'TN': '221208_110756:cmsbot_crab_20221208_120755', 'dbStatus': u'SUBMITTED', 'testResult': 'TestRunning', 'jobsPerStatus': {'idle': 1, 'finished': 4}, 'combinedStatus': 'SUBMITTED'}
{'TN': '221208_110757:cmsbot_crab_20221208_120756', 'dbStatus': u'SUBMITTED', 'testResult': 'TestFailed', 'jobsPerStatus': {'failed': 1}, 'combinedStatus': 'FAILED'}
{'TN': '221208_110758:cmsbot_crab_20221208_120758', 'dbStatus': u'SUBMITTED', 'testResult': 'TestFailed', 'jobsPerStatus': {'failed': 1, 'finished': 9}, 'combinedStatus': 'FAILED'}
{'TN': '221208_110800:cmsbot_crab_20221208_120759', 'dbStatus': u'SUBMITTED', 'testResult': 'TestFailed', 'jobsPerStatus': {'failed': 1, 'finished': 98, 'transferring': 1}, 'combinedStatus': 'SUBMITTED'}
{'TN': '221208_110827:cmsbot_crab_20221208_120801', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 10}, 'combinedStatus': 'COMPLETED'}
cmsbuild commented 1 year ago

Test: Task Submission Status Tracking Result: FULL-STATUS-UNKNOWN Attempt: 1 out of 4. Will run again. Finished at: 2022-12-08 14:13:27 Test log: https://cmssdt.cern.ch/dmwm-jenkins/job/CRABServer_CheckTestResults/411/console

{'TN': '221208_110754:cmsbot_crab_20221208_120754', 'dbStatus': u'SUBMITTED', 'testResult': 'TestFailed', 'jobsPerStatus': {'failed': 56, 'finished': 944}, 'combinedStatus': 'FAILED'}
{'TN': '221208_110756:cmsbot_crab_20221208_120755', 'dbStatus': u'SUBMITTED', 'testResult': 'TestRunning', 'jobsPerStatus': {'finished': 5, 'running': 2}, 'combinedStatus': 'SUBMITTED'}
{'TN': '221208_110757:cmsbot_crab_20221208_120756', 'dbStatus': u'SUBMITTED', 'testResult': 'TestFailed', 'jobsPerStatus': {'failed': 1}, 'combinedStatus': 'FAILED'}
{'TN': '221208_110758:cmsbot_crab_20221208_120758', 'dbStatus': u'SUBMITTED', 'testResult': 'TestFailed', 'jobsPerStatus': {'failed': 1, 'finished': 9}, 'combinedStatus': 'FAILED'}
{'TN': '221208_110800:cmsbot_crab_20221208_120759', 'dbStatus': u'SUBMITTED', 'testResult': 'TestFailed', 'jobsPerStatus': {'failed': 2, 'finished': 98}, 'combinedStatus': 'FAILED'}
{'TN': '221208_110827:cmsbot_crab_20221208_120801', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 10}, 'combinedStatus': 'COMPLETED'}
cmsbuild commented 1 year ago

Test: Task Submission Status Tracking Result: FAILED Attempt: 2 out of 4. Test failed. Investigate manually Finished at: 2022-12-08 15:15:47 Test log: https://cmssdt.cern.ch/dmwm-jenkins/job/CRABServer_CheckTestResults/413/console

{'TN': '221208_110754:cmsbot_crab_20221208_120754', 'dbStatus': u'SUBMITTED', 'testResult': 'TestFailed', 'jobsPerStatus': {'failed': 56, 'finished': 944}, 'combinedStatus': 'FAILED'}
{'TN': '221208_110756:cmsbot_crab_20221208_120755', 'dbStatus': u'SUBMITTED', 'testResult': 'TestFailed', 'jobsPerStatus': {'failed': 2, 'finished': 6, 'running': 1}, 'combinedStatus': 'SUBMITTED'}
{'TN': '221208_110757:cmsbot_crab_20221208_120756', 'dbStatus': u'SUBMITTED', 'testResult': 'TestFailed', 'jobsPerStatus': {'failed': 1}, 'combinedStatus': 'FAILED'}
{'TN': '221208_110758:cmsbot_crab_20221208_120758', 'dbStatus': u'SUBMITTED', 'testResult': 'TestFailed', 'jobsPerStatus': {'failed': 1, 'finished': 9}, 'combinedStatus': 'FAILED'}
{'TN': '221208_110800:cmsbot_crab_20221208_120759', 'dbStatus': u'SUBMITTED', 'testResult': 'TestFailed', 'jobsPerStatus': {'failed': 2, 'finished': 98}, 'combinedStatus': 'FAILED'}
{'TN': '221208_110827:cmsbot_crab_20221208_120801', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 10}, 'combinedStatus': 'COMPLETED'}