search

dmwm / DAS

Data Aggregation System
11 stars 7 forks source link

Removing obsolete datasets #4264

Closed dfehling closed 8 years ago

dfehling commented 8 years ago

Hello,

I know that individual users are able to go in and invalidate their obsolete and removed datasets, but would it be possible to set up an automated process that runs once a month, or once a week that does the same?

I'm trying to run over a very old dataset:

https://cmsweb.cern.ch/das/request?view=list&limit=50&instance=prod%2Fphys02&input=dataset%3D%2FTT_Mtt-1000toInf_CT10_scaledown_TuneZ2star_8TeV-powheg-tauola%2Fjpilot-Summer12_DR53X-PU_S10_START53_V7A-v1_TLBSM_53x_v2-c04f3b4fa74c8266c913b71e0c74901d%2FUSER

and it's listed as VALID. However, if I checked the lpctlbsm area at FNAL, there does not appear to anything located in that path. I think DAS knows that because a site search:

https://cmsweb.cern.ch/das/request?input=site%20dataset%3D/TT_Mtt-1000toInf_CT10_scaledown_TuneZ2star_8TeV-powheg-tauola/jpilot-Summer12_DR53X-PU_S10_START53_V7A-v1_TLBSM_53x_v2-c04f3b4fa74c8266c913b71e0c74901d/USER&instance=prod/phys02&idx=0&limit=10

gives the following:

Site: N/A WARNING: "combined service unable to process your request", click on show link to get more info Site type: TAPE no user access, StorageElement: N/A SiteDB Sources: combined show Site: T1_US_FNAL_MSS Datasets, SiteDB Sources: dbs3 show

I reached out to the FNAL administrators and they confirm that the dataset is not located on their disk.

Is there some way to set up an automated job that would automatically invalidate deleted datasets?

Thanks, Dave

vkuznet commented 8 years ago

Dave, it is not a question to DAS, first DAS does not hold any data, it asks other CMS services for that. Second, the dataset information is stored in DBS database and therefore such request should be addressed over there. Finally, the dataset invalidation is a task of data-ops group. Therefore I would suggest you post your question to appropriate HN, e.g. hn-cms-dataopsrequests@cern.ch hn-cms-dmDevelopment@cern.ch

Best, Valentin.

On 0, dfehling notifications@github.com wrote:

Hello,

I know that individual users are able to go in and invalidate their obsolete and removed datasets, but would it be possible to set up an automated process that runs once a month, or once a week that does the same?

I'm trying to run over a very old dataset:

https://cmsweb.cern.ch/das/request?view=list&limit=50&instance=prod%2Fphys02&input=dataset%3D%2FTT_Mtt-1000toInf_CT10_scaledown_TuneZ2star_8TeV-powheg-tauola%2Fjpilot-Summer12_DR53X-PU_S10_START53_V7A-v1_TLBSM_53x_v2-c04f3b4fa74c8266c913b71e0c74901d%2FUSER

and it's listed as VALID. However, if I checked the lpctlbsm area at FNAL, there does not appear to anything located in that path. I think DAS knows that because a site search:

https://cmsweb.cern.ch/das/request?input=site%20dataset%3D/TT_Mtt-1000toInf_CT10_scaledown_TuneZ2star_8TeV-powheg-tauola/jpilot-Summer12_DR53X-PU_S10_START53_V7A-v1_TLBSM_53x_v2-c04f3b4fa74c8266c913b71e0c74901d/USER&instance=prod/phys02&idx=0&limit=10

gives the following:

Site: N/A WARNING: "combined service unable to process your request", click on show link to get more info Site type: TAPE no user access, StorageElement: N/A SiteDB Sources: combined show Site: T1_US_FNAL_MSS Datasets, SiteDB Sources: dbs3 show

I reached out to the FNAL administrators and they confirm that the dataset is not located on their disk.

Is there some way to set up an automated job that would automatically invalidate deleted datasets?

Thanks, Dave

You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/dmwm/DAS/issues/4264