search

dmwm / CRABServer

15 stars 37 forks source link

Shoud T0_CH_CERN_Disk not be considered as a valid disk location? #4970

Closed AndresTanasijczuk closed 8 years ago

AndresTanasijczuk commented 8 years ago

See https://hypernews.cern.ch/HyperNews/CMS/get/computing-tools/895.html

According to DAS https://cmsweb.cern.ch/das/request?input=site%20dataset%3D/ZeroBias/Run2015C-LumiPixelsMinBias-PromptReco-v1/ALCARECO&instance=prod/global&idx=0&limit=10 dataset is in T0_CH_CERN_Disk (partially), T1_US_FNAL_Buffer (partially) and T1_US_FNAL_MSS (partially). CRAB considers T0_CH_CERN_Disk as a valid disk location: https://github.com/dmwm/CRABServer/blob/3.3.1507.rc11/src/python/TaskWorker/Actions/DBSDataDiscovery.py#L39

juztas commented 8 years ago

mh, If I remember correctly T0_CH_CERN_Disk was added to AAA on 2015-07-20, so it is valid, but better if they use only T2_CH_CERN for running jobs

mmascher commented 8 years ago

Hi Andres,

I would also like to understand why the error message comes from splitter.py and is kind of misleading:

splitting task 150825_013756:capalmer_crab_PCC_ZeroBias_DataCert_WeekLater_150820
on dataset /ZeroBias/Run2015C-LumiPixelsMinBias-PromptReco-v1/ALCARECO
with LumiBased method does not generate any job

probably a separate issue though

belforte commented 8 years ago

Better check with Dirk Sent from Stefano's mobile On Aug 26, 2015 09:57, Andres Tanasijczuk notifications@github.com wrote:See https://hypernews.cern.ch/HyperNews/CMS/get/computing-tools/895.html

According to DAS https://cmsweb.cern.ch/das/request?input=site%20dataset%3D/ZeroBias/Run2015C-LumiPixelsMinBias-PromptReco-v1/ALCARECO&instance=prod/global&idx=0&limit=10 dataset is in T0_CH_CERN_Disk (partially), T1_US_FNAL_Buffer (partially) and T1_US_FNAL_MSS (partially). CRAB considers T0_CH_CERN_Disk as a valid disk location: https://github.com/dmwm/CRABServer/blob/3.3.1507.rc11/src/python/TaskWorker/Actions/DBSDataDiscovery.py#L39

—Reply to this email directly or view it on GitHub.

belforte commented 8 years ago

@hufnagel Can you say what's the usage for T0_CH_CERN_Disk ? Is that phedex node going to say ? And if yes, which data will it hold ? And in case should we consider it a valid location for directing CRAB jobs at (hopefully via a proper PSN assigned to it in SiteDb which is not there now https://cmsweb.cern.ch/sitedb/prod/sites/T0_CH_CERN). I expect that we should ignore T0_CH_CERN_Disk , but wanted to be sure.

belforte commented 8 years ago

labelled as bug because in any case the error message is wrong, as also Marco pointed out, given what's currently in SiteDB this should have appeared as a dataset with no accessible location (no PSN).

hufnagel commented 8 years ago

T0_CH_CERN_Disk is the Tier0 production disk. DataOps is currently evaluating running workflows there as well.

As far as samples there goes, they are available via AAA, but Tier0 samples only stay there as long as they are needed for internal Tier0 operations, then they are deleted. Usually that means any piece of data is resident on T0_CH_CERN_Disk for a few days at most. Same for future DataOps production, although the time window during which the data is resident would likely be longer.

The associated PSN is currently only usable for production jobs, CRAB should not submit to it.

As far as allowing users to submit CRAB jobs that read via AAA from T0_CH_CERN_Disk, this is in principle possible, but OTOH given the short time window data is on T0_CH_CERN_Disk, might be more trouble than its worth. At the moment its completely pointless as he disk is only used for short lived Tier0 output, so you are better off ignoring T0_CH_CERN_Disk.

belforte commented 8 years ago

thanks Dirk. Then we could stay with current situation where T0_CH_CERN_Disk has no associated PSN in SiteDB and thus CRAB will ignore it.