search

dmwm / DAS

Data Aggregation System
11 stars 7 forks source link

Panic in `dasgoclient` #4279

Closed schneiml closed 5 years ago

schneiml commented 6 years ago

I am trying to find some RelVal MC files. With my query and dasgoclient, it tells me I need the lumi parameter:

$ dasgoclient --query 'file run=1 dataset=/RelValTTbar_NLO_Pow_13TeV_py8/CMSSW_10_0_0_pre3-100X_upgrade2018_realistic_v4_gen-v1/GEN-SIM'
[{"error":"DBS unable to unmarshal the data into DAS record, api=file4DatasetRunLumi, data={\"exception\": 400, \"message\": \"files API does not supprt run_num=1 when no lumi.\", \"type\": \"HTTPError\"}, error=json: cannot unmarshal object into Go value of type []mongo.DASRecord"}]

But if I pass the lumi parameter, I get a Panic:

$ dasgoclient --query 'file run=1 dataset=/RelValTTbar_NLO_Pow_13TeV_py8/CMSSW_10_0_0_pre3-100X_upgrade2018_realistic_v4_gen-v1/GEN-SIM lumi=1'
panic: interface conversion: interface {} is nil, not []interface {}

goroutine 1 [running]:
github.com/dmwm/das2go/services.LocalAPIs.L_dbs3_file4dataset_run_lumi(0xc420098380, 0x7d, 0x7fff9c7fa431, 0x77, 0xc4202542a0, 0x20, 0xc420368a50, 0xc4201c9830, 0x1, 0x1, ...)
    /afs/cern.ch/work/v/valya/builds/w630/BUILD/slc6_amd64_gcc630/cms/dasgoclient-binary/v01.01.03/dasgoclient/gopath/src/github.com/dmwm/das2go/services/dbs.go:239 +0x610
github.com/dmwm/das2go/services.(*LocalAPIs).L_dbs3_file4dataset_run_lumi(0xaaa680, 0xc420098380, 0x7d, 0x7fff9c7fa431, 0x77, 0xc4202542a0, 0x20, 0xc420368a50, 0xc4201c9830, 0x1, ...)
    <autogenerated>:1 +0x6c
reflect.Value.call(0x85de00, 0xaaa680, 0x2e93, 0x867e72, 0x4, 0xc4201c3088, 0x1, 0x1, 0xc4201c3178, 0x859420, ...)
    /usr/lib/golang/src/reflect/value.go:434 +0x905
reflect.Value.Call(0x85de00, 0xaaa680, 0x2e93, 0xc4201c3088, 0x1, 0x1, 0xaaa680, 0x2e93, 0x1)
    /usr/lib/golang/src/reflect/value.go:302 +0xa4
main.processLocalApis(0xc420098380, 0x7d, 0x7fff9c7fa431, 0x77, 0xc4202542a0, 0x20, 0xc420368a50, 0xc4201c9830, 0x1, 0x1, ...)
    /afs/cern.ch/work/v/valya/builds/w630/BUILD/slc6_amd64_gcc630/cms/dasgoclient-binary/v01.01.03/dasgoclient/main.go:693 +0x505
main.process(0x7fff9c7fa431, 0x77, 0x0, 0x867ab0, 0x1, 0x0, 0x0, 0x0, 0x86e48e, 0x16, ...)
    /afs/cern.ch/work/v/valya/builds/w630/BUILD/slc6_amd64_gcc630/cms/dasgoclient-binary/v01.01.03/dasgoclient/main.go:345 +0x3176
main.main()
    /afs/cern.ch/work/v/valya/builds/w630/BUILD/slc6_amd64_gcc630/cms/dasgoclient-binary/v01.01.03/dasgoclient/main.go:93 +0x6d9

das_client accepts the query without lumi, but I am still waiting for a result. das_client with the lumi parameter gives the expected results.

vkuznet commented 6 years ago

Marcel, it is complex issue and you're not using right syntax in this case. Due to DBS API restriction this type of queries we do in one shot like this: file,run,lumi dataset=/RelValTTbar_NLO_Pow_13TeV_py8/CMSSW_10_0_0_pre3-100X_upgrade2018_realistic_v4_gen-v1/GEN-SIM

For instance:

dasgoclient -query "file,run,lumi dataset=/RelValTTbar_NLO_Pow_13TeV_py8/CMSSW_10_0_0_pre3-100X_upgrade2018_realistic_v4_gen-v1/GEN-SIM" /store/relval/CMSSW_10_0_0_pre3/RelValTTbar_NLO_Pow_13TeV_py8/GEN-SIM/100X_upgrade2018_realistic_v4_gen-v1/20000/32117F52-40EE-E711-BCC3-0CC47A4C8F0A.root 1 [34,38,37,36] /store/relval/CMSSW_10_0_0_pre3/RelValTTbar_NLO_Pow_13TeV_py8/GEN-SIM/100X_upgrade2018_realistic_v4_gen-v1/20000/30849043-40EE-E711-B196-0025905B8574.root 1 [19,23,21,14,18,30,32] /store/relval/CMSSW_10_0_0_pre3/RelValTTbar_NLO_Pow_13TeV_py8/GEN-SIM/100X_upgrade2018_realistic_v4_gen-v1/20000/24E759C9-3FEE-E711-81FC-0CC47A78A2EC.root 1 [11,40,22,13] /store/relval/CMSSW_10_0_0_pre3/RelValTTbar_NLO_Pow_13TeV_py8/GEN-SIM/100X_upgrade2018_realistic_v4_gen-v1/20000/2851BE4B-40EE-E711-9716-0CC47A4D7692.root 1 [24,7,16,10,20,25,26] /store/relval/CMSSW_10_0_0_pre3/RelValTTbar_NLO_Pow_13TeV_py8/GEN-SIM/100X_upgrade2018_realistic_v4_gen-v1/20000/2AAE8666-3FEE-E711-A3E4-0CC47A7C34A6.root 1 [6,9,15,3,1,2,5] /store/relval/CMSSW_10_0_0_pre3/RelValTTbar_NLO_Pow_13TeV_py8/GEN-SIM/100X_upgrade2018_realistic_v4_gen-v1/20000/587C89EC-41EE-E711-A679-0CC47A7C35E0.root 1 [12] /store/relval/CMSSW_10_0_0_pre3/RelValTTbar_NLO_Pow_13TeV_py8/GEN-SIM/100X_upgrade2018_realistic_v4_gen-v1/20000/447BA6C7-40EE-E711-B939-0CC47A4C8F06.root 1 [28,8,17,39,29] /store/relval/CMSSW_10_0_0_pre3/RelValTTbar_NLO_Pow_13TeV_py8/GEN-SIM/100X_upgrade2018_realistic_v4_gen-v1/20000/88CC31F7-40EE-E711-A5AA-0CC47A7C340C.root 1 [35,4,27,31,33]

Then you can find your file based on output. All DAS interfaces should support this query (DAS web, DAS python client and DAS go client).

Best, Valentin.

On 0, Marcel Schneider notifications@github.com wrote:

I am trying to find some RelVal MC files. With my query and dasgoclient, it tells me I need the lumi parameter:

$ dasgoclient --query 'file run=1 dataset=/RelValTTbar_NLO_Pow_13TeV_py8/CMSSW_10_0_0_pre3-100X_upgrade2018_realistic_v4_gen-v1/GEN-SIM'
[{"error":"DBS unable to unmarshal the data into DAS record, api=file4DatasetRunLumi, data={\"exception\": 400, \"message\": \"files API does not supprt run_num=1 when no lumi.\", \"type\": \"HTTPError\"}, error=json: cannot unmarshal object into Go value of type []mongo.DASRecord"}]

But if I pass the lumi parameter, I get a Panic:

$ dasgoclient --query 'file run=1 dataset=/RelValTTbar_NLO_Pow_13TeV_py8/CMSSW_10_0_0_pre3-100X_upgrade2018_realistic_v4_gen-v1/GEN-SIM lumi=1'
panic: interface conversion: interface {} is nil, not []interface {}

goroutine 1 [running]:
github.com/dmwm/das2go/services.LocalAPIs.L_dbs3_file4dataset_run_lumi(0xc420098380, 0x7d, 0x7fff9c7fa431, 0x77, 0xc4202542a0, 0x20, 0xc420368a50, 0xc4201c9830, 0x1, 0x1, ...)
  /afs/cern.ch/work/v/valya/builds/w630/BUILD/slc6_amd64_gcc630/cms/dasgoclient-binary/v01.01.03/dasgoclient/gopath/src/github.com/dmwm/das2go/services/dbs.go:239 +0x610
github.com/dmwm/das2go/services.(*LocalAPIs).L_dbs3_file4dataset_run_lumi(0xaaa680, 0xc420098380, 0x7d, 0x7fff9c7fa431, 0x77, 0xc4202542a0, 0x20, 0xc420368a50, 0xc4201c9830, 0x1, ...)
  <autogenerated>:1 +0x6c
reflect.Value.call(0x85de00, 0xaaa680, 0x2e93, 0x867e72, 0x4, 0xc4201c3088, 0x1, 0x1, 0xc4201c3178, 0x859420, ...)
  /usr/lib/golang/src/reflect/value.go:434 +0x905
reflect.Value.Call(0x85de00, 0xaaa680, 0x2e93, 0xc4201c3088, 0x1, 0x1, 0xaaa680, 0x2e93, 0x1)
  /usr/lib/golang/src/reflect/value.go:302 +0xa4
main.processLocalApis(0xc420098380, 0x7d, 0x7fff9c7fa431, 0x77, 0xc4202542a0, 0x20, 0xc420368a50, 0xc4201c9830, 0x1, 0x1, ...)
  /afs/cern.ch/work/v/valya/builds/w630/BUILD/slc6_amd64_gcc630/cms/dasgoclient-binary/v01.01.03/dasgoclient/main.go:693 +0x505
main.process(0x7fff9c7fa431, 0x77, 0x0, 0x867ab0, 0x1, 0x0, 0x0, 0x0, 0x86e48e, 0x16, ...)
  /afs/cern.ch/work/v/valya/builds/w630/BUILD/slc6_amd64_gcc630/cms/dasgoclient-binary/v01.01.03/dasgoclient/main.go:345 +0x3176
main.main()
  /afs/cern.ch/work/v/valya/builds/w630/BUILD/slc6_amd64_gcc630/cms/dasgoclient-binary/v01.01.03/dasgoclient/main.go:93 +0x6d9

das_client accepts the query without lumi, but I am still waiting for a result.

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/dmwm/DAS/issues/4279

vkuznet commented 6 years ago

But it is still a bug and I'll fix it, please keep issue open.

On 0, Marcel Schneider notifications@github.com wrote:

I am trying to find some RelVal MC files. With my query and dasgoclient, it tells me I need the lumi parameter:

$ dasgoclient --query 'file run=1 dataset=/RelValTTbar_NLO_Pow_13TeV_py8/CMSSW_10_0_0_pre3-100X_upgrade2018_realistic_v4_gen-v1/GEN-SIM'
[{"error":"DBS unable to unmarshal the data into DAS record, api=file4DatasetRunLumi, data={\"exception\": 400, \"message\": \"files API does not supprt run_num=1 when no lumi.\", \"type\": \"HTTPError\"}, error=json: cannot unmarshal object into Go value of type []mongo.DASRecord"}]

But if I pass the lumi parameter, I get a Panic:

$ dasgoclient --query 'file run=1 dataset=/RelValTTbar_NLO_Pow_13TeV_py8/CMSSW_10_0_0_pre3-100X_upgrade2018_realistic_v4_gen-v1/GEN-SIM lumi=1'
panic: interface conversion: interface {} is nil, not []interface {}

goroutine 1 [running]:
github.com/dmwm/das2go/services.LocalAPIs.L_dbs3_file4dataset_run_lumi(0xc420098380, 0x7d, 0x7fff9c7fa431, 0x77, 0xc4202542a0, 0x20, 0xc420368a50, 0xc4201c9830, 0x1, 0x1, ...)
  /afs/cern.ch/work/v/valya/builds/w630/BUILD/slc6_amd64_gcc630/cms/dasgoclient-binary/v01.01.03/dasgoclient/gopath/src/github.com/dmwm/das2go/services/dbs.go:239 +0x610
github.com/dmwm/das2go/services.(*LocalAPIs).L_dbs3_file4dataset_run_lumi(0xaaa680, 0xc420098380, 0x7d, 0x7fff9c7fa431, 0x77, 0xc4202542a0, 0x20, 0xc420368a50, 0xc4201c9830, 0x1, ...)
  <autogenerated>:1 +0x6c
reflect.Value.call(0x85de00, 0xaaa680, 0x2e93, 0x867e72, 0x4, 0xc4201c3088, 0x1, 0x1, 0xc4201c3178, 0x859420, ...)
  /usr/lib/golang/src/reflect/value.go:434 +0x905
reflect.Value.Call(0x85de00, 0xaaa680, 0x2e93, 0xc4201c3088, 0x1, 0x1, 0xaaa680, 0x2e93, 0x1)
  /usr/lib/golang/src/reflect/value.go:302 +0xa4
main.processLocalApis(0xc420098380, 0x7d, 0x7fff9c7fa431, 0x77, 0xc4202542a0, 0x20, 0xc420368a50, 0xc4201c9830, 0x1, 0x1, ...)
  /afs/cern.ch/work/v/valya/builds/w630/BUILD/slc6_amd64_gcc630/cms/dasgoclient-binary/v01.01.03/dasgoclient/main.go:693 +0x505
main.process(0x7fff9c7fa431, 0x77, 0x0, 0x867ab0, 0x1, 0x0, 0x0, 0x0, 0x86e48e, 0x16, ...)
  /afs/cern.ch/work/v/valya/builds/w630/BUILD/slc6_amd64_gcc630/cms/dasgoclient-binary/v01.01.03/dasgoclient/main.go:345 +0x3176
main.main()
  /afs/cern.ch/work/v/valya/builds/w630/BUILD/slc6_amd64_gcc630/cms/dasgoclient-binary/v01.01.03/dasgoclient/main.go:93 +0x6d9

das_client accepts the query without lumi, but I am still waiting for a result.

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/dmwm/DAS/issues/4279

vkuznet commented 6 years ago

I applied fix into das2go: https://github.com/dmwm/das2go/commit/110d95c670e710027877d6dc0125fb190de92163 Now I need to test it and pass to CMSSW.

vkuznet commented 6 years ago

Submitted to CMSSW, see https://github.com/cms-sw/cmsdist/pull/3674

schneiml commented 6 years ago

Thanks!

The query you suggested also seems to work with [1], which is what I need; it is just inconvenient that I need to pass the run number for real data, while i must not pass a run number for MC. But that is not a real problem.

Edit: Wait, no, it does not. The addtional output columns end up in the file names. I can use a fixed list of files for now, but I'll have to look at that again...

[1] https://github.com/cms-sw/cmssw/blob/master/Configuration/Applications/python/ConfigBuilder.py#L130

vkuznet commented 5 years ago

Marcel, could you please check again if this issue is valid. I applied fixes to dasgoclient.

schneiml commented 5 years ago

Hi Valentin, I have not look at this in a long time, which probably means it worked well enough... I'll get back to it in a few days, we might need this code for some Heavy Ions preparations.

vkuznet commented 5 years ago

No response from end-user, will assume that issue was fixed, since appropriate fix was submitted to CMSSW. If necessary please re-open and provide new example

schneiml commented 5 years ago

Indeed, works fine for what I need atm.