search

dmwm / das2go

Go implementation of Data Aggregation System (DAS) for CMS experiment
MIT License
2 stars 3 forks source link

DAS dataset showing 0 size/files/events when files exist #60

Closed borzari closed 12 months ago

borzari commented 12 months ago

I am not sure if I should ask this here or in some ticket or cms-talk. Please, tell me otherwise.

I have some datasets registered on DAS that show 0 files, size or events, even though the files exist somewhere. As an example, dataset /DYJetsToLL_M-50_TuneCP5_13p6TeV-madgraphMLM-pythia8/borzari-Run3Summer22EEDRPremix-124X_mcRun3_2022_realistic_postEE_v1-v3_EXODisappTrkPAT-9d47b08148d63ab5a7c705d3cebd063d/USER has files in /store/user/borzari/DYJetsToLL_M-50_TuneCP5_13p6TeV-madgraphMLM-pythia8/Run3Summer22EEDRPremix-124X_mcRun3_2022_realistic_postEE_v1-v3_EXODisappTrkPAT/230827_180418/0000/, but the DAS information shows 0 files.

I also have the same information if I use dasgoclient -query 'dataset dataset=/DYJetsToLL_M-50_TuneCP5_13p6TeV-madgraphMLM-pythia8/borzari-Run3Summer22EEDRPremix-124X_mcRun3_2022_realistic_postEE_v1-v3_EXODisappTrkPAT-9d47b08148d63ab5a7c705d3cebd063d/USER' -json.

I noticed that this usually happens for older datasets, because when this dataset was created, DAS showed the correct information associated with it.

Is this behavior of changing the registered information to 0 an expected feature? Is there a way to revert this and put the initial information back in DAS?

vkuznet commented 12 months ago

I think the issue is that those files were invalidated at some point, and I think @belforte knows why and can explain the situation.

Meanwhile your files are there, e.g.

./dasgoclient -query="file dataset=/DYJetsToLL_M-50_TuneCP5_13p6TeV-madgraphMLM-pythia8/borzari-Run3Summer22EEDRPremix-124X_mcRun3_2022_realistic_postEE_v1-v3_EXODisappTrkPAT-9d47b08148d63ab5a7c705d3cebd063d/USER status=* instance=prod/phys03"
...

You only need to supply status=* option to your query which provide access to ALL files, valid and invalid, while by default DAS only shows valid files.

Said that, it is not issue with DAS, and rather invalidation of files in DBS and I suggest you ask this question in cms-talk forum.

belforte commented 12 months ago

@borzari please see https://cms-talk.web.cern.ch/t/dbs-phys03-files-wrongly-invalidated/31152/2

borzari commented 12 months ago

Thank you @vkuznet for the answer and suggestion!

And thank you @belforte as well for the solution! The command seems to have worked.

Just a quick question: will the correct size/files/events information be displayed when searching for the dataset at some point? This isn't that important, but would be good to have the information available again for the datasets that have the files validity modified (asking because the dataset information is still 0, but the files query worked fine).

vkuznet commented 12 months ago

May be I was not clear, with status=* flag DAS shows both valid and invalid files, in your case all files are invalid. Therefore, without this flags DAS will still show 0 files unless someone in operations will change these files back to valid status. When and if it will be done the DAS will show again valid files.