#49: Tests for prod CRABClient using preprod REST instance and CMSSW_8_0_36 CMSSW release started.

cmsbuild commented 3 years ago

Tests started for following configuration:

Configuration:

CRABClient_version: prod
- REST_Instance: preprod
- CMSSW_release: CMSSW_8_0_36
- SCRAM_ARCH: slc7_amd64_gcc530

Tests started:

Task_Submission_Status_Tracking: true
Client_Validation_Suite: true
Client_Configuration_Validation: true

Link to the job

cmsbuild commented 3 years ago

Task submission for Task_Submission_Status_Tracking successfully ended. Task submission for Client_Configuration_Validation successfully ended. Task submission for Client_Validation_Suite successfully ended. Find submission log here

cmsbuild commented 3 years ago

Client configuration validation results: TEST-FAILED: Some CRABClient commands failed. Find failed commands below:

TEST_COMMAND: crab createmyproxy --proxy=/tmp/x509up_u501 --days=100
..................................................................................................TEST_RESULT: [FAILED] TEST_MESSAGE: Checking credentials Your identity: /DC=ch/DC=cern/OU=Organic Units/OU=Users/CN=cmsbot/CN=545661/CN=Robot: CMS Build Bot Creating temporary proxy Done Contacting voms2.cern.ch:15002 [/DC=ch/DC=cern/OU=computers/CN=voms2.cern.ch] "cms" Done Creating proxy Done

Your proxy is valid until Tue Oct 26 15:14:02 2021 Registering user credentials for server /DC=ch/DC=cern/OU=computers/CN=crab-(preprod|prod|dev)-tw(02|01).cern.ch|/DC=ch/DC=cern/OU=computers/CN=(ddi|ddidk|mytw).cern.ch|/DC=ch/DC=cern/OU=computers/CN=stefanov(m|m2).cern.ch Problems delegating My-proxy. It seems your proxy has not been delegated to myproxy. Please check the logfile for the exact error (Maybe you simply typed a wrong password)

TEST_COMMAND: crab report --outputdir=/build/cmsbld/jenkins/workspace/COPYCAT_CRABServer_ClientValidation/logdir --proxy=/tmp/x509up_u501 --dir=crab_20211019_105106
TEST_RESULT: [FAILED] TEST_MESSAGE: Running crab status first to fetch necessary information. Problems delegating My-proxy. Error trying to create credential: Problems delegating My-proxy. It seems your proxy has not been delegated to myproxy. Please check the logfile for the exact error (Maybe you simply typed a wrong password) Error trying to create credential: Problems delegating My-proxy. It seems your proxy has not been delegated to myproxy. Please check the logfile for the exact error (it might simply you typed a wrong password)

TEST_COMMAND: crab getlog --quantity=2 --outputpath=/build/cmsbld/jenkins/workspace/COPYCAT_CRABServer_ClientValidation/logdir --dump --jobids=2,3 --checksum=no --proxy=/tmp/x509up_u501 --dir=crab_20211019_105106
TEST_RESULT: [FAILED] TEST_MESSAGE: Problems delegating My-proxy. Error trying to create credential: Problems delegating My-proxy. It seems your proxy has not been delegated to myproxy. Please check the logfile for the exact error (Maybe you simply typed a wrong password) Error trying to create credential: Problems delegating My-proxy. It seems your proxy has not been delegated to myproxy. Please check the logfile for the exact error (it might simply you typed a wrong password)

TEST_COMMAND: crab getoutput --parallel=10 --wait=4 --outputpath=/build/cmsbld/jenkins/workspace/COPYCAT_CRABServer_ClientValidation/logdir --jobids=2,3 --checksum=yes --proxy=/tmp/x509up_u501 --dir=crab_20211019_105106
TEST_RESULT: [FAILED] TEST_MESSAGE: Problems delegating My-proxy. Error trying to create credential: Problems delegating My-proxy. It seems your proxy has not been delegated to myproxy. Please check the logfile for the exact error (Maybe you simply typed a wrong password) Error trying to create credential: Problems delegating My-proxy. It seems your proxy has not been delegated to myproxy. Please check the logfile for the exact error (it might simply you typed a wrong password)

TEST_COMMAND: crab getoutput --parallel=10 --wait=4 --outputpath=/build/cmsbld/jenkins/workspace/COPYCAT_CRABServer_ClientValidation/logdir --jobids=2,3 --checksum=no --proxy=/tmp/x509up_u501 --dir=crab_20211019_105106
TEST_RESULT: [FAILED] TEST_MESSAGE: Problems delegating My-proxy. Error trying to create credential: Problems delegating My-proxy. It seems your proxy has not been delegated to myproxy. Please check the logfile for the exact error (Maybe you simply typed a wrong password) Error trying to create credential: Problems delegating My-proxy. It seems your proxy has not been delegated to myproxy. Please check the logfile for the exact error (it might simply you typed a wrong password)

TEST_COMMAND: crab getoutput --parallel=10 --wait=4 --outputpath=/build/cmsbld/jenkins/workspace/COPYCAT_CRABServer_ClientValidation/logdir --dump --jobids=2,3 --checksum=yes --proxy=/tmp/x509up_u501 --dir=crab_20211019_105106
TEST_RESULT: [FAILED] TEST_MESSAGE: Problems delegating My-proxy. Error trying to create credential: Problems delegating My-proxy. It seems your proxy has not been delegated to myproxy. Please check the logfile for the exact error (Maybe you simply typed a wrong password) Error trying to create credential: Problems delegating My-proxy. It seems your proxy has not been delegated to myproxy. Please check the logfile for the exact error (it might simply you typed a wrong password)

TEST_COMMAND: crab getoutput --parallel=10 --wait=4 --outputpath=/build/cmsbld/jenkins/workspace/COPYCAT_CRABServer_ClientValidation/logdir --dump --jobids=2,3 --checksum=no --proxy=/tmp/x509up_u501 --dir=crab_20211019_105106
TEST_RESULT: [FAILED] TEST_MESSAGE: Problems delegating My-proxy. Error trying to create credential: Problems delegating My-proxy. It seems your proxy has not been delegated to myproxy. Please check the logfile for the exact error (Maybe you simply typed a wrong password) Error trying to create credential: Problems delegating My-proxy. It seems your proxy has not been delegated to myproxy. Please check the logfile for the exact error (it might simply you typed a wrong password)

TEST_COMMAND: crab getoutput --parallel=10 --wait=4 --outputpath=/build/cmsbld/jenkins/workspace/COPYCAT_CRABServer_ClientValidation/logdir --xrootd --jobids=2,3 --checksum=yes --proxy=/tmp/x509up_u501 --dir=crab_20211019_105106
TEST_RESULT: [FAILED] TEST_MESSAGE: Problems delegating My-proxy. Error trying to create credential: Problems delegating My-proxy. It seems your proxy has not been delegated to myproxy. Please check the logfile for the exact error (Maybe you simply typed a wrong password) Error trying to create credential: Problems delegating My-proxy. It seems your proxy has not been delegated to myproxy. Please check the logfile for the exact error (it might simply you typed a wrong password)

TEST_COMMAND: crab getoutput --parallel=10 --wait=4 --outputpath=/build/cmsbld/jenkins/workspace/COPYCAT_CRABServer_ClientValidation/logdir --xrootd --jobids=2,3 --checksum=no --proxy=/tmp/x509up_u501 --dir=crab_20211019_105106
TEST_RESULT: [FAILED] TEST_MESSAGE: Problems delegating My-proxy. Error trying to create credential: Problems delegating My-proxy. It seems your proxy has not been delegated to myproxy. Please check the logfile for the exact error (Maybe you simply typed a wrong password) Error trying to create credential: Problems delegating My-proxy. It seems your proxy has not been delegated to myproxy. Please check the logfile for the exact error (it might simply you typed a wrong password)

TEST_COMMAND: crab getoutput --quantity=1 --parallel=10 --wait=4 --outputpath=/build/cmsbld/jenkins/workspace/COPYCAT_CRABServer_ClientValidation/logdir --checksum=yes --proxy=/tmp/x509up_u501 --dir=crab_20211019_105106
TEST_RESULT: [FAILED] TEST_MESSAGE: Problems delegating My-proxy. Error trying to create credential: Problems delegating My-proxy. It seems your proxy has not been delegated to myproxy. Please check the logfile for the exact error (Maybe you simply typed a wrong password) Error trying to create credential: Problems delegating My-proxy. It seems your proxy has not been delegated to myproxy. Please check the logfile for the exact error (it might simply you typed a wrong password)

TEST_COMMAND: crab getoutput --quantity=3 --parallel=10 --wait=4 --outputpath=/build/cmsbld/jenkins/workspace/COPYCAT_CRABServer_ClientValidation/logdir --checksum=no --proxy=/tmp/x509up_u501 --dir=crab_20211019_105106
TEST_RESULT: [FAILED] TEST_MESSAGE: Problems delegating My-proxy. Error trying to create credential: Problems delegating My-proxy. It seems your proxy has not been delegated to myproxy. Please check the logfile for the exact error (Maybe you simply typed a wrong password) Error trying to create credential: Problems delegating My-proxy. It seems your proxy has not been delegated to myproxy. Please check the logfile for the exact error (it might simply you typed a wrong password)

Finished at: 2021-10-19 11:14:53 Find test log here. Find all executed commands here.

cmsbuild commented 3 years ago

Client configuration validation results:

RETRY TESTS: transferLogs-check.sh 211019_085110:cmsbot_crab_transferLogs - 2 outputFiles-check.sh 211019_085115:cmsbot_crab_outputFiles - 2 maxMemoryMB-check.sh 211019_085118:cmsbot_crab_maxMemoryMB - 2 maxJobRuntimeMin-check.sh 211019_085119:cmsbot_crab_maxJobRuntimeMin - 2 numCores-check.sh 211019_085121:cmsbot_crab_numCores - 2 scriptExe-check.sh 211019_085122:cmsbot_crab_scriptExe - 2 useParent-check.sh 211019_085131:cmsbot_crab_useParent - 2 lumiMaskFile-check.sh 211019_085134:cmsbot_crab_lumiMaskFile - 2 lumiMaskUrl-check.sh 211019_085135:cmsbot_crab_lumiMaskUrl - 2 runRange-check.sh 211019_085137:cmsbot_crab_runRange - 2 outLFNDirBase-check.sh 211019_085139:cmsbot_crab_outLFNDirBase - 2 whitelist-check.sh 211019_085143:cmsbot_crab_whitelist - 2 ignoreGlobalBlacklist-check.sh 211019_085145:cmsbot_crab_ignoreGlobalBlacklist - 2 extraJDL-check.sh 211019_085149:cmsbot_crab_extraJDL - 2

SUCCESSFUL TESTS: inputFiles-check.sh 211019_085113:cmsbot_crab_inputFiles - 0 allowUndistributedCMSSW-check.sh 211019_085117:cmsbot_crab_allowUndistributedCMSSW - 0 sendPythonFolder-check.sh 211019_085126:cmsbot_crab_sendPythonFolder - 0 sendExternalFolder-check.sh 211019_085129:cmsbot_crab_sendExternalFolder - 0 inputDBS-check.sh 211019_085130:cmsbot_crab_inputDBS - 0 scheddName-check.sh 211019_085147:cmsbot_crab_scheddName - 0 collector-check.sh 211019_085148:cmsbot_crab_collector - 0

FAILED TESTS: transferOutputs-check.sh 211019_085109:cmsbot_crab_transferOutputs - 1 activity-check.sh 211019_085111:cmsbot_crab_activity - 1 disableAutomaticOutputCollection-check.sh 211019_085114:cmsbot_crab_disableAutomaticOutputCollection - 1 scriptArgs-check.sh 211019_085123:cmsbot_crab_scriptArgs - 1 secondaryInputDataset-check.sh 211019_085133:cmsbot_crab_secondaryInputDataset - 1 ignoreLocality-check.sh 211019_085140:cmsbot_crab_ignoreLocality - 1 userInputFiles-check.sh 211019_085141:cmsbot_crab_userInputFiles - 1 blacklist-check.sh 211019_085144:cmsbot_crab_blacklist - 1

Finished at: 2021-10-19 11:14:53 Find test log here.

cmsbuild commented 3 years ago

Task status tracking results:

FULL-STATUS-UNKNOWN: Status checking will be executed again because some jobs are still running

{'TN': '211019_085152:cmsbot_crab_20211019_105150', 'dbStatus': u'SUBMITTED', 'testResult': 'TestRunning', 'jobsPerStatus': {'finished': 940, 'transferring': 48, 'running': 12}, 'combinedStatus': 'SUBMITTED'} {'TN': '211019_085154:cmsbot_crab_20211019_105153', 'dbStatus': u'SUBMITTED', 'testResult': 'TestFailed', 'jobsPerStatus': {'failed': 2, 'running': 1, 'finished': 2}, 'combinedStatus': 'SUBMITTED'} {'TN': '211019_085157:cmsbot_crab_20211019_105155', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 1}, 'combinedStatus': 'COMPLETED'} {'TN': '211019_085200:cmsbot_crab_20211019_105158', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 10}, 'combinedStatus': 'COMPLETED'} {'TN': '211019_085202:cmsbot_crab_20211019_105200', 'dbStatus': u'SUBMITTED', 'testResult': 'TestRunning', 'jobsPerStatus': {'finished': 92, 'transferring': 8}, 'combinedStatus': 'SUBMITTED'} {'TN': '211019_085206:cmsbot_crab_20211019_105202', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 10}, 'combinedStatus': 'COMPLETED'}

Finished at: 2021-10-19 11:52:32 Find test log here.

cmsbuild commented 3 years ago

Task status tracking results:

TESTS-FAILED: There is at least one task where job execution was not completed successfully. Please investigate task manually and start tests again.

{'TN': '211019_085152:cmsbot_crab_20211019_105150', 'dbStatus': u'SUBMITTED', 'testResult': 'TestFailed', 'jobsPerStatus': {'failed': 5, 'finished': 991, 'transferring': 3, 'running': 1}, 'combinedStatus': 'SUBMITTED'} {'TN': '211019_085154:cmsbot_crab_20211019_105153', 'dbStatus': u'SUBMITTED', 'testResult': 'TestFailed', 'jobsPerStatus': {'failed': 3, 'finished': 2, 'running': 1}, 'combinedStatus': 'SUBMITTED'} {'TN': '211019_085157:cmsbot_crab_20211019_105155', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 1}, 'combinedStatus': 'COMPLETED'} {'TN': '211019_085200:cmsbot_crab_20211019_105158', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 10}, 'combinedStatus': 'COMPLETED'} {'TN': '211019_085202:cmsbot_crab_20211019_105200', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 100}, 'combinedStatus': 'COMPLETED'} {'TN': '211019_085206:cmsbot_crab_20211019_105202', 'dbStatus': u'SUBMITTED', 'testResult': 'TestPassed', 'jobsPerStatus': {'finished': 10}, 'combinedStatus': 'COMPLETED'}

Finished at: 2021-10-19 12:53:03 Find test log here.

ddaina / variousStuff

#49: Tests for prod CRABClient using preprod REST instance and CMSSW_8_0_36 CMSSW release started. #41