Data location and update for ACDC requests

dmwm / WMCore

Core workflow management components for CMS.

Apache License 2.0

46 stars 107 forks source link

Data location and update for ACDC requests #7307

Open paorozo opened 8 years ago

paorozo commented 8 years ago

For https://its.cern.ch/jira/browse/CMSCOMPPR-147 (cerminar_Run2016D-v2-HTMHT-23Sep2016_8020_160926_165944_8575), we decided to replicate the input dataset to Nebraska and FNAL by hand, when the Rereco workflow was still running.
When the assistance time came, we ACDC with FNAL and Nebraska in whitelist only.
Then we realised the ACDCs were sitting in acquired. e.g. for the DataProcessing task https://cmsweb.cern.ch/couchdb/workqueue/_design/WorkQueue/_rewrite/element/ed121cbfadcb166f9e5ba59d1ee995c1, the ACDC only knows that the input dataset is in T1_FR_CCIN2P3,T1_UK_RAL.
Is there something we can do about this?

Thank you.

amaltaro commented 8 years ago

Hmmm, I thought the data location would be updated for ACDCs as well. For the moment, can you please use the data location of the inputs as reported in the ACDCServer docs (+ the new sites you want to have it running)?

This is one more ACDC case that requires a review.

amaltaro commented 8 years ago

BTW, if you don't mind I'm going to update the subject of this issue.

paorozo commented 8 years ago

Please, go ahead. Thanks for your reply.

amaltaro commented 7 years ago

I took a look at this code in the past week when Jean-Roch was asking a few questions. We do not update data location for ACDC "inputs". Each WQE has a Inputs parameter with either the block, dataset or empty for MC. For ACDC requests, it has the collection name, and for those cases we don't update data location at GQ nor at LQ: https://github.com/dmwm/WMCore/blob/master/src/python/WMCore/WorkQueue/DataLocationMapper.py#L174

The way forward would be to assign your workflow to the new location and the old one (location in the docs). Otherwise there will be no match between site whitelist and data location.

amaltaro commented 7 years ago

Data location and update for ACDC requests

amaltaro commented 6 years ago

@prozober given that ACDCs can be made either on unmerged or merged files, doing a phedex/dbs data location lookup doesn't seem right. If you ever need to read acdc files remotely, please use the TrustSitelists flag.