search

dmwm / das2go

Go implementation of Data Aggregation System (DAS) for CMS experiment
MIT License
2 stars 3 forks source link

Accessing information via DAS WebInterface and dasgoclient #29

Closed wajidalikhan closed 3 years ago

wajidalikhan commented 3 years ago

Dear Colleagues,

I am trying to access the following information: dataset=/BTagMu/Run2018A-12Nov2019_UL2018-v1/MINIAOD | grep dataset.nevents, dataset.nblocks, dataset.nfiles

and it displays the desired result: Number of blocks: 77 Number of events: 22633270 Number of files: 480

But there are a coupld of datasets sets for which I need this information. But using the das client it dasgoclient -query="dataset=/BTagMu/Run2018A-12Nov2019_UL2018-v1/MINIAOD | grep dataset.nevents, dataset.nblocks, dataset.nfiles"

Unable to extract filters=[dataset [0] nevents], error=Key path not found Unable to extract filters=[dataset [0] nblocks], error=Key path not found Unable to extract filters=[dataset [0] nfiles], error=Key path not found Unable to extract filters=[dataset [0] nevents], error=Key path not found Unable to extract filters=[dataset [0] nblocks], error=Key path not found Unable to extract filters=[dataset [0] nfiles], error=Key path not found Unable to extract filters=[dataset [0] nevents], error=Key path not found Unable to extract filters=[dataset [0] nblocks], error=Key path not found Unable to extract filters=[dataset [0] nfiles], error=Key path not found

22633270 77 480

at the end it gives the desired result printed. A quick search revels the fact that we do that query, it returns three results - from dbs3:dataset_info, dbs3:datasetlist and dbs3:filesummaries. Only the third one has nevetns, nblocks, nfiles. First two don't have them, so you get two set of errors about "Key path not found" and then the result that you want.

Could you please give your expert opinion how can we filter on dbs3:filesummaries so that we don't have those filters issues when using the dasgoclient.

Regards, Wajid

vkuznet commented 3 years ago

Thanks for reporting the error, I fixed it in this commit https://github.com/dmwm/dasgoclient/commit/3cf4452eacf4131955fe35babd87f5195adf5565 and will schedule new dasgoclient to CMSSW build probably later today or tomorrow. Once build you'll no longer get the error message in non-verbose mode. I'll provide update about CMSSW ticket when it will be available

wajidalikhan commented 3 years ago

Great, thanks a lot for your quick action.

vkuznet commented 3 years ago

Here is CMSSW PR: https://github.com/cms-sw/cmsdist/pull/6387, usually once it is merged the new dasgoclient appears in cvmfs. Anyway you can check its version to be sure. Here is a new one:

./dasgoclient -version
Build: git=v02.04.02 go=go1.13.8 date=2020-11-09 20:38:22.076625567 +0100 CET m=+0.004636402
vkuznet commented 3 years ago

The final PR is https://github.com/cms-sw/cmsdist/pull/6401 and it is merged now. Therefore new dasgoclient version will appear soon in CMSSW, the new version of dasgoclient is v02.04.04. Everything is done here, closing the ticket

wajidalikhan commented 3 years ago

Thanks a lot. The new version should also appear here, right: /cvmfs/cms.cern.ch/common/dasgoclient

at moment it still seems the previous version. Build: git=v02.03.02 go=go1.13.11 date=2020-11-12 04:35:52.669366213 +0100 CET m=+0.011022451

vkuznet commented 3 years ago

it appeared only today, see

/cvmfs/cms.cern.ch/common/dasgoclient -version
Build: git=v02.04.04 go=go1.13.8 date=2020-11-13 14:47:23.467936739 +0100 CET m=+0.005653588