CMS: transfers with rucio

katilp commented 3 years ago

Setup

source /cvmfs/cms.cern.ch/cmsset_default.sh
source /cvmfs/cms.cern.ch/rucio/setup-py3.sh
voms-proxy-init -voms cms -rfc -valid 192:00
export RUCIO_ACCOUNT=`whoami`

Quota

Check:

rucio list-account-limits $RUCIO_ACCOUNT

Add (as manager):

rucio-admin account set-limits $RUCIO_ACCOUNT T3_CH_CERN_OpenData 20TB

Transfer

Start:

$ rucio add-rule cms:/DoubleMuon/Run2015D-16Dec2015-v1/MINIAOD 1 T3_CH_CERN_OpenData
cdce9cca6ad34b1d938fd74d9e9e557a

Check:

$ rucio list-rules --account $RUCIO_ACCOUNT
ID                                ACCOUNT    SCOPE:NAME                                                                      STATE[OK/REPL/STUCK]    RSE_EXPRESSION         COPIES  EXPIRES (UTC)    CREATED (UTC)
--------------------------------  ---------  ------------------------------------------------------------------------------  ----------------------  -------------------  --------  ---------------  -------------------
cdce9cca6ad34b1d938fd74d9e9e557a  kati       cms:/DoubleMuon/Run2015D-16Dec2015-v1/MINIAOD                                   REPLICATING[0/1068/0]   T3_CH_CERN_OpenData         1                   2021-07-26 07:07:27

or

$ rucio rule-info cdce9cca6ad34b1d938fd74d9e9e557a
Id:                         cdce9cca6ad34b1d938fd74d9e9e557a
Account:                    kati
Scope:                      cms
Name:                       /DoubleMuon/Run2015D-16Dec2015-v1/MINIAOD
RSE Expression:             T3_CH_CERN_OpenData
Copies:                     1
State:                      REPLICATING
Locks OK/REPLICATING/STUCK: 0/1068/0
Grouping:                   DATASET
Expires at:                 None
Locked:                     False
Weight:                     None
Created at:                 2021-07-26 07:07:27
Updated at:                 2021-07-26 07:07:30
Error:                      None
Subscription Id:            None
Source replica expression:  None
Activity:                   User Subscriptions
Comment:                    None
Ignore Quota:               False
Ignore Availability:        False
Purge replicas:             False
Notification:               NO
End of life:                None
Child Rule Id:              None

Monitor

https://cms-rucio-webui.cern.ch/rule?rule_id=cdce9cca6ad34b1d938fd74d9e9e557a

https://monit-kibana.cern.ch/kibana/goto/c3656cd9856f234d2c241e3a34f6f8b8

Datasets at eospublic

Check if they land in a proper location

export EOS_MGM_URL=root://eospublic.cern.ch
eos ls -lh /eos/opendata/cms/

katilp commented 3 years ago

The first test transfer done!

Next MC:

$ rucio add-rule cms:/DYJetsToLL_M-10to50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/RunIIFall15MiniAODv2-PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v1/MINIAODSIM 1 T3_CH_CERN_OpenData
68a557d5fd5147f6bb7df66d755edb1f

$ rucio rule-info 68a557d5fd5147f6bb7df66d755edb1f
Id:                         68a557d5fd5147f6bb7df66d755edb1f
Account:                    kati
Scope:                      cms
Name:                       /DYJetsToLL_M-10to50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/RunIIFall15MiniAODv2-PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v1/MINIAODSIM
RSE Expression:             T3_CH_CERN_OpenData
Copies:                     1
State:                      REPLICATING
Locks OK/REPLICATING/STUCK: 0/753/0
Grouping:                   DATASET
Expires at:                 None
Locked:                     False
Weight:                     None
Created at:                 2021-08-13 15:20:32
Updated at:                 2021-08-13 15:20:34
Error:                      None
Subscription Id:            None
Source replica expression:  None
Activity:                   User Subscriptions
Comment:                    None
Ignore Quota:               False
Ignore Availability:        False
Purge replicas:             False
Notification:               NO
End of life:                None
Child Rule Id:              None

katilp commented 3 years ago

[x] Transfer 2015 collision MINIAOD datasets (total size: 14TB )
- /DoubleMuon/Run2015D-16Dec2015-v1/MINIAOD transferred already as test
[x] Transfer 2015 MC MINIAODSIM datasets (total size: should be 227TB)
- /DYJetsToLL_M-10to50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/RunIIFall15MiniAODv2-PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v1/MINIAODSIM transferred already as test
[x] Transfer 2015 collision AOD datasets (total size: 117TB)

Monitoring: https://cms-rucio-webui.cern.ch/r2d2 (Since October, add the username in the URL)

katilp commented 3 years ago

Some errors when creating MC rules of type:

403440d015514f2b8414bd22ad20916d
2021-09-20 14:35:33,328 ERROR   An unknown exception occurred.
Details: no error information passed (http status code: 502 ('bad_gateway',))
0aa486cf78ed472cad393f2f4cf1b09b
--
0115d7d6a6384da881739e47d51f477d
2021-09-20 14:57:25,780 ERROR   An unknown exception occurred.
Details: no error information passed (http status code: 504 ('gateway_timeout',))
3ae40ff057bf45a68aabf590b34924b3

27 of such errors for datasets:

/DStarToD0Pi_D0K3Pi_DStarFilter_TuneCUEP8M1_13TeV-pythia8-evtgen/RunIIFall15MiniAODv2-PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v1/MINIAODSIM
/DYJetsToLL_M-50_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/RunIIFall15MiniAODv2-PU25nsData2015v1_HCALDebug_76X_mcRun2_asymptotic_v12-v1/MINIAODSIM
/DYJetsToLL_M-5to50_TuneCUETP8M1_13TeV-madgraphMLM-pythia8/RunIIFall15MiniAODv2-PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v1/MINIAODSIM
/DYJetsToQQ_HT180_13TeV-madgraphMLM-pythia8/RunIIFall15MiniAODv2-PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v1/MINIAODSIM
/DoubleElectron_FlatPt-300To6500/RunIIFall15MiniAODv2-PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v1/MINIAODSIM
/DoublePhotonNoMaterial_FlatPt-100To6500/RunIIFall15MiniAODv2-PU25nsData2015v1_NoMaterial_76X_mcRun2_asymptotic_v12-v1/MINIAODSIM
/GluGluToHHTo4B_node_4_13TeV-madgraph/RunIIFall15MiniAODv2-PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v1/MINIAODSIM
/MinBias_TuneCUETHS1-13TeV-herwigpp/RunIIFall15MiniAODv2-PU25nsData2015v1_castor_76X_mcRun2_asymptotic_v12-v1/MINIAODSIM
/QCD_HT100to200_BGenFilter_TuneCUETP8M1_13TeV-madgraphMLM-pythia8/RunIIFall15MiniAODv2-PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v1/MINIAODSIM
/QCD_Pt-120to170_EMEnriched_TuneCUETP8M1_13TeV_pythia8/RunIIFall15MiniAODv2-PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v1/MINIAODSIM
/QCD_Pt-15TTo7000_TuneZ2star-Flat_13TeV_pythia6/RunIIFall15MiniAODv2-PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v1/MINIAODSIM
/QCD_Pt-80to120_MuEnrichedPt5_TuneCUETP8M1_13TeV_pythia8/RunIIFall15MiniAODv2-PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v1/MINIAODSIM
/QCD_Pt_250toInf_bcToE_TuneCUETP8M1_13TeV_pythia8/RunIIFall15MiniAODv2-PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v1/MINIAODSIM
/QCD_Pt_30to80_bcToE_TuneCUETP8M1_13TeV_pythia8/RunIIFall15MiniAODv2-PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v2/MINIAODSIM
/ST_t-channel_4f_scaledown_leptonDecays_13TeV-amcatnlo-pythia8_TuneCUETP8M1/RunIIFall15MiniAODv2-PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v1/MINIAODSIM
/SingleNeutron_E200to500/RunIIFall15MiniAODv2-PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v1/MINIAODSIM
/TTJets_HT-800to1200_TuneCUETP8M1_13TeV-madgraphMLM-pythia8/RunIIFall15MiniAODv2-PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext1-v1/MINIAODSIM
/TTJets_SingleLeptFromT_TuneCUETP8M1_13TeV-madgraphMLM-pythia8/RunIIFall15MiniAODv2-PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v1/MINIAODSIM
/TTToSemiLeptonic_13TeV-powheg/RunIIFall15MiniAODv2-PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext1-v1/MINIAODSIM
/WJetsToLNu_HT-100To200_TuneCUETP8M1_13TeV-madgraphMLM-pythia8/RunIIFall15MiniAODv2-PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v1/MINIAODSIM
/ZJetsToNuNu_HT-600To800_13TeV-madgraph/RunIIFall15MiniAODv2-PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v1/MINIAODSIM
/ZJetsToNuNu_HT-600ToInf_13TeV-madgraph/RunIIFall15MiniAODv2-PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext1-v1/MINIAODSIM
/ttWJets_13TeV_madgraphMLM/RunIIFall15MiniAODv2-PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v1/MINIAODSIM
/TT_TuneCUETP8M1_13TeV-powheg-pythia8/RunIIFall15MiniAODv2-PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext3-v1/MINIAODSIM
/TT_TuneCUETP8M1_13TeV-powheg-pythia8/RunIIFall15MiniAODv2-PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext4-v1/MINIAODSIM
/ZToEE_NNPDF30_13TeV-powheg_M_2300_3500/RunIIFall15MiniAODv2-PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v1/MINIAODSIM
/ZToEE_NNPDF30_13TeV-powheg_M_3500_4500/RunIIFall15MiniAODv2-PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v1/MINIAODSIM

Out of that list only 3 were missing and now done:

7c55ded9c00d4cdfa714d1259e5fc1cc /ZJetsToNuNu_HT-600ToInf_13TeV-madgraph/RunIIFall15MiniAODv2-PU25nsData2015v1_76X_mcRun2_asymptotic_v12_ext1-v1/MINIAODSIM
1ffa4044f1fe480183471c1a83f48973 /ZToEE_NNPDF30_13TeV-powheg_M_2300_3500/RunIIFall15MiniAODv2-PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v1/MINIAODSIM
6180acd0d2614bd1ba763103108e1c89 /ZToEE_NNPDF30_13TeV-powheg_M_3500_4500/RunIIFall15MiniAODv2-PU25nsData2015v1_76X_mcRun2_asymptotic_v12-v1/MINIAODSIM

Unclear how the transfers got done with the errors. Maybe the request still got through but did not output the rule id. But now looks that all MC done (tbc).

katilp commented 3 years ago

Errors on AOD transfer:

/Tau/Run2015D-16Dec2015-v1/AOD
f8b92f263a514c73aff4e502d14fc2a6
/SinglePhoton/Run2015D-16Dec2015-v1/AOD
071ec06e25664ad9b5c862cebce90cc8
/SingleMuon/Run2015D-16Dec2015-v1/AOD
299caa151c46460bb125ce852f16b18d
/SingleElectron/Run2015D-08Jun2016-v1/AOD
0ebe1ed51b06439eb670fe51080bcaad
/MuonEG/Run2015D-16Dec2015-v1/AOD
55b15abd840349f7ac69485fcca9ca17
/MuOnia/Run2015D-16Dec2015-v1/AOD
58bbce71e1904d2e876c4862167075be
/MET/Run2015D-16Dec2015-v1/AOD
2021-10-06 09:43:15,025 ERROR   An unknown exception occurred.
Details: no error information passed (http status code: 504 ('gateway_timeout',))
/JetHT/Run2015D-16Dec2015-v1/AOD
2021-10-06 09:44:15,812 ERROR   An unknown exception occurred.
Details: no error information passed (http status code: 504 ('gateway_timeout',))
/HTMHT/Run2015D-16Dec2015-v1/AOD
2021-10-06 09:45:16,589 ERROR   An unknown exception occurred.
Details: no error information passed (http status code: 504 ('gateway_timeout',))
/DoubleMuonLowMass/Run2015D-16Dec2015-v1/AOD
c88d96da491944588a7d2067edbcf417
/DoubleMuon/Run2015D-16Dec2015-v1/AOD
2021-10-06 09:46:23,729 ERROR   An unknown exception occurred.
Details: no error information passed (http status code: 504 ('gateway_timeout',))
/DisplacedJet/Run2015D-16Dec2015-v1/AOD
2021-10-06 09:47:24,502 ERROR   An unknown exception occurred.
Details: no error information passed (http status code: 504 ('gateway_timeout',))
/Charmonium/Run2015D-16Dec2015-v1/AOD
2021-10-06 09:48:25,406 ERROR   An unknown exception occurred.
Details: no error information passed (http status code: 504 ('gateway_timeout',))
/BTagMu/Run2015D-16Dec2015-v1/AOD
2021-10-06 09:49:26,238 ERROR   An unknown exception occurred.
Details: no error information passed (http status code: 504 ('gateway_timeout',))
/BTagCSV/Run2015D-16Dec2015-v1/AOD
92e25f09140f49dc843ad728c7f686c3
/DoubleEG/Run2015D-08Jun2016-v1/AOD
2021-10-06 09:49:41,845 ERROR   There is not enough quota left to fulfil the operation.
Details: There is insufficient quota on any of the target RSE's to fullfill the operation.
/ZeroBias/Run2015D-16Dec2015-v1/AOD
2021-10-06 09:49:42,773 ERROR   There is not enough quota left to fulfil the operation.
Details: There is insufficient quota on any of the target RSE's to fullfill the operation.

Out of gateway timeout errors only BTagMu did not start, others are OK.

Increased quota to 400TB (the current limit counts the existing stuff + the new one). Add the failed ones:

/BTagMu/Run2015D-16Dec2015-v1/AOD
5fec0ea4df6a4841a3fc78b0eb2cea71
/DoubleEG/Run2015D-08Jun2016-v1/AOD
b111078a82554831a325f05ad0798969
/ZeroBias/Run2015D-16Dec2015-v1/AOD
6c89d03b85a8462292063e324176b431

katilp commented 2 years ago

For the record, a rucio account t3_ch_cern_opendata_local exists and maybe the next transfers should be done using it. Its quota is set to the total of /eospublic/cms.

cernopendata / data-curation