Add possibility to use bearer tokens instead of X509 proxy

oshadura commented 3 years ago

Users at coffea-casa (https://coffea-casa.readthedocs.io/en/latest/) prototype of CMS analysis facility requested to add dasgoclient as one of the tools available for users.

At our facility, we are not using X509 proxy anymore, but its future replacement - bearer tokens: I added here a link to a description of its discovery specification: https://github.com/WLCG-AuthZ-WG/bearer-token-discovery/blob/master/specification.md

Would it be possible to add support for the discovery of bearer tokens? Thank you in advance! (cc: @bbockelm)

vkuznet commented 3 years ago

Oksana, I added support for token authorization via these commits: https://github.com/dmwm/dasgoclient/commit/276815777c10a9300b9e1c288fe596211a11f08e and https://github.com/dmwm/das2go/commit/db59f8af8f5b4ff0760684091399e0701f5f8948 and new version of the dsagoclient v02.04.29 supports it (feel free to grab this version from release area).

But in order to use token based authentication we need to upgrade cmsweb infrastructure. This is on-going effort and it will take a while since it depends on many cms services to complete the transition. Meanwhile, you may test new client using https://cmsweb-auth.cern.ch URL (so far it is only visible within CERN network and, therefore, you'll need to run it either form lxplus or create proper tunnel). For that you need few steps:

# adjust existing dasmaps to use cmsweb-auth.cern.ch url
# edit the following file $HOME/.dasmaps/das_maps_dbs_prod.js and replace
# cmsweb.cern.ch:8443 to cmsweb-auth.cern.ch

# obtain new access token from https://cmsweb-auth.cern.ch/token and place it into /tmp/token file

# run dasgoclient with new token and new host url
dasgoclient -query ... -host https://cmsweb-auth.cern.ch -token /tmp/token -verbose 3

If you'll run new dasgoclient in verbose 3 mode you'll see that it will properly put token into HTTP header and place request to cmsweb-auth.cern.ch to obtain the data using this token.

oshadura commented 3 years ago

Thanks a lot for such a quick reply! I will try to test it and give you feedback...

Checking commits, it seems for me it doesn't support BEARER_TOKEN or BEARER_TOKEN_FILE env variables, right? It will be nice if client will support them (e.g. for example we are using them at coffea-casa and I added the same in ROOT https://github.com/root-project/root/pull/7068)

vkuznet commented 3 years ago

With this commit https://github.com/dmwm/dasgoclient/commit/7eda26b22e8bbe44f5f4411d52a8ce970566c227 it is there. Feel free to use v02.04.30 release version.

oshadura commented 3 years ago

Thanks! I will test it and let you know if it will work in our environment.

oshadura commented 3 years ago

@vkuznet I tried locally to test it, but it didn't work for me (?). I generated token using oidc-agent from https://opensciencegrid.org/technology/software/requesting-tokens/ for https://cmsweb-auth.cern.ch (see coffea-casa issuer). I got approved @ https://cmsweb-auth.cern.ch: oidc-agent:coffea-casa-macksu.dyndns.cern.ch The device has been approved.

cat ~/.config/oidc-agent/issuer.config
https://wlcg.cloud.cnaf.infn.it/ WLCG
https://cms-auth.web.cern.ch/ coffea-casa%

and then I copy/pasted token in BEARER_TOKEN env variable as well in /tmp/token:

~  export BEARER_TOKEN="............."
~ ./dasgoclient_osx -query="dataset=/ZMM*/*/*" 
Neither X509_USER_PROXY or X509_USER_KEY/X509_USER_CERT are set
In order to run please obtain valid proxy via "voms-proxy-init -voms cms -rfc"
and setup X509_USER_PROXY or setup X509_USER_KEY/X509_USER_CERT in your environment

Didn't work for me either adding -host https://cmsweb-auth.cern.ch -token /tmp/token:

./dasgoclient_osx -host https://cmsweb-auth.cern.ch -token /tmp/token -verbose 3
Neither X509_USER_PROXY or X509_USER_KEY/X509_USER_CERT are set
In order to run please obtain valid proxy via "voms-proxy-init -voms cms -rfc"
and setup X509_USER_PROXY or setup X509_USER_KEY/X509_USER_CERT in your environment

CC: @bbockelm

vkuznet commented 3 years ago

Oksana, sorry about that, I forgot that x509 is forced in das server by default (and my env settings always have X509). I made appropriate changes in this commit https://github.com/dmwm/das2go/commit/21731d5e912544d49249f663d305323afce43469 and made new version of dasgoclient with it. Please take new release version v02.04.31 which should solve the problem.

And, even though is not related to token per-se, you will need to add to your env DBS_URL=https://cmsweb-auth.cern.ch to avoid default queries to go to cmsweb. I run the test on lxplus and it seems to be fine, but I still get one DBS error (ERROR: das exit with code: 2 , error: DBS unable to unmarshal the data into DAS record, api=datasetlist, data=Hello /datasetlist, error=invalid character 'H' looking for beginning of value) which is not related to this issue and therefore it can be ignored.

bbockelm commented 3 years ago

@oshadura - just to make sure you and Valentin are on the same page, can you post the public part of the token payload (as decoded by JWT.io). That'll help in debugging if you get permission denied.

vkuznet commented 3 years ago

Oksana, since I didn't use for really long time DBS_URL to point to other instances, I overlooked what should be supplied in this var. Finally, I got time and it should be set in this case to export DBS_URL=https://cmsweb-auth.cern.ch/dbs/int/global/DBSReader. This eliminates the DBS error I mentioned before. With this setup and corrected ~/.dasmaps/das_maps_dbs_prod.js file (to use cmsweb-auth.cern.ch instead of cmsweb.cern.ch:8443) now everything runs fine in my local setup.

I also want to clarify that cmsweb-auth is not for production usage as we use it as testbed for tokens based requests. And, in order to migrate dasgoclient to token based auth we'll need first migrate cmsweb as a whole. Therefore, even after tests are successful the coffea-casa users may need to wait for this transition to happen, otherwise we'll need to maintain cmsweb-auth or similar frontend to provide token based access to cmsweb services.

oshadura commented 3 years ago

@bbockelm my payload data is: { "wlcg.ver": "1.0", "sub": "dd67f011-4018-4aa1-9350-d7c84245dc86", "aud": "https://wlcg.cern.ch/jwt/v1/any", "nbf": 1617794648, "scope": "address storage.create:/ phone openid offline_access profile storage.read:/ eduperson_scoped_affiliation storage.modify:/ eduperson_entitlement email wlcg", "iss": "https://cms-auth.web.cern.ch/", "exp": 1617798248, "iat": 1617794648, "jti": "31370fd1-6e93-4404-81d8-4e2bacb374e6", "client_id": "cfedb4b7-4491-4ecb-9998-e6b85580285e" }

oshadura commented 3 years ago

@vkuznet we have hidden all settings from users so, in the end, I am sure we will be able to switch transparently. coffea-casa is still a prototype of the analysis facility so we have an opportunity to test new features earlier.

vkuznet commented 3 years ago

@bbockelm Brian, it would be extremely useful to get oidc-agent on CC7 (better to be installed on lxplus). I tried instructions from here and CC7 repo is not accessible, the instructions for CC7 lack of proof as I'm getting the following error (after installing all dependencies):

make
/usr/bin/ld: cannot find -llist
collect2: error: ld returned 1 exit status
make: *** [bin/oidc-agent] Error 1

I'm not sure where I should pass this info and hope you can give me some directions.

Bottom line, of course rpm/deb packages are nice, but it would be even better just to get static executable for Linux and avoid the avalanche of dependencies.

oshadura commented 3 years ago

@vkuznet I wanted to ask you, did you test from lxplus? I still didn't manage to make it work on local MacOS machine.

On AF I tried to build it as well, and it still didn't work for me. Am I am doing something wrong?

jovyan@jupyter-oksana-2eshadura-40cern-2ech:~/dasgoclient$ make build_all
sed -i -e "s,{{VERSION}},v02.04.31,g" main.go
go clean; rm -rf pkg dasgoclient_osx; GOOS=darwin go build -ldflags="-s -w -extldflags -static"
go: downloading github.com/dmwm/das2go v0.0.0-20210407122922-21731d5e9125
go: downloading github.com/pkg/profile v1.5.0
go: downloading github.com/buger/jsonparser v1.1.1
go: downloading gopkg.in/mgo.v2 v2.0.0-20190816093944-a6b53ec6cb22
go: downloading github.com/vkuznet/dcr v0.0.0-20200922173840-46cf19c474b3
go: downloading github.com/vkuznet/x509proxy v0.0.0-20191014143623-163039704c67
sed -i -e "s,v02.04.31,{{VERSION}},g" main.go
mv dasgoclient dasgoclient_osx
sed -i -e "s,{{VERSION}},v02.04.31,g" main.go
go clean; rm -rf pkg dasgoclient_amd64; GOOS=linux go build -ldflags="-s -w -extldflags -static"
sed -i -e "s,v02.04.31,{{VERSION}},g" main.go
mv dasgoclient dasgoclient_amd64
sed -i -e "s,{{VERSION}},v02.04.31,g" main.go
go clean; rm -rf pkg dasgoclient_ppc64le; GOARCH=ppc64le GOOS=linux go build -ldflags="-s -w -extldflags -static"
sed -i -e "s,v02.04.31,{{VERSION}},g" main.go
mv dasgoclient dasgoclient_ppc64le
sed -i -e "s,{{VERSION}},v02.04.31,g" main.go
go clean; rm -rf pkg dasgoclient_aarch64; GOARCH=arm64 GOOS=linux go build -ldflags="-s -w -extldflags -static"
sed -i -e "s,v02.04.31,{{VERSION}},g" main.go
mv dasgoclient dasgoclient_aarch64
go clean; rm -rf pkg dasgoclient.exe; GOARCH=amd64 GOOS=windows go build -ldflags="-s -w -extldflags -static"

jovyan@jupyter-oksana-2eshadura-40cern-2ech:~/dasgoclient$ ./dasgoclient_amd64 -host https://cmsweb-auth.cern.ch -token /etc/cmsaf-secrets/xcache_token -verbose 3
Neither X509_USER_PROXY or X509_USER_KEY/X509_USER_CERT are set
In order to run please obtain valid proxy via "voms-proxy-init -voms cms -rfc"
and setup X509_USER_PROXY or setup X509_USER_KEY/X509_USER_CERT in your environment
jovyan@jupyter-oksana-2eshadura-40cern-2ech:~/dasgoclient$ env | grep BEARER_TOKEN
BEARER_TOKEN_FILE=/etc/cmsaf-secrets/xcache_token

vkuznet commented 3 years ago

Oksana, it seems that I forgot to commit local changes for go.mod which cause build to pick up old dependencies. I just redid everything (with v02.04.33 tag), and here is my commands on lxplus:

# unset all X509 env's
 unset X509_USER_CERT
 unset X509_USER_KEY
 unset X509_USER_PROXY

# get dasgoclient
curl -ksLO https://github.com/dmwm/dasgoclient/releases/download/v02.04.33/dasgoclient_amd64
chmod +x dasgoclient_amd64
# get token
cat > /tmp/valya/token
....
# setup DBS_URL
export DBS_URL=https://cmsweb-auth.cern.ch
# changed dasmpas
vim ~/.dasmaps/das_maps_dbs_prod.js
# run dasgoclient
./dasgoclient_amd64 -query "dataset=/ZMM*/*/*" -host https://cmsweb-auth.cern.ch -token /tmp/valya/token -verbose 3
...
/ZMM/Summer11-DESIGN42_V11_428_SLHC1-v1/GEN-SIM
/ZMM_13TeV_TuneCP5-pythia8/RunIIAutumn18DR-SNBHP_SNB_HP_102X_upgrade2018_realistic_v17-v2/AODSIM
...

bbockelm commented 3 years ago

@vkuznet - would you be able to try the oidc-agent from the OSG repositories? That's what I've been using.

The OSG keeps a few notes on how to use oidc-agent to get WLCG (and, with a few changes, CMS) tokens here:

https://opensciencegrid.org/technology/software/requesting-tokens/

oshadura commented 3 years ago

@vkuznet looks it almost works! Thank you so much! I was trying to use your example...

jovyan@jupyter-oksana-2eshadura-40cern-2ech:~/dasgoclient$ ./dasgoclient_amd64 -query "dataset=/ZMM*/*/*" -json -host https://cmsweb-auth.cern.ch -token /etc/cmsaf-secrets/xcache_token -verbose 3
DBSUrl:  https://cmsweb-auth.cern.ch/dbs/prod/DBSReader
SitedbUrl:  https://cmsweb.cern.ch:8443/sitedb/data/prod
CricUrl w/ site API:  https://cms-cric.cern.ch/api/cms/site/query
RucioUrl:  http://cms-rucio.cern.ch
RucioAuthUrl:  https://cms-rucio-auth.cern.ch/auth/x509
Load dasmaps /home/jovyan/.dasmaps/das_maps_dbs_prod.js
2021/04/09 10:36:47 process, idx 0, val dataset, nval =, nnval /ZMM*/*/*
2021/04/09 10:36:47 ERROR DAS QL ERROR, query=dataset = /ZMM*/*/*, idx=0, msg=Wrong DAS key: dataset
DAS QL ERROR, query=dataset = /ZMM*/*/*, idx=0, msg=Wrong DAS key: dataset
dataset=/ZMM*/*/*
^
ERROR: das parser error: DAS QL ERROR, query=dataset = /ZMM*/*/*, idx=0, msg=Wrong DAS key: dataset
jovyan@jupyter-oksana-2eshadura-40cern-2ech:~/dasgoclient$

vkuznet commented 3 years ago

Oksana, this error usually means that you do not have dasmaps in place or the file is corrupted. Could you please check that you have this file $HOME/.dasmaps/das_maps_dbs_prod.js and check its content. This file is downloaded from this URL https://raw.githubusercontent.com/dmwm/DASMaps/master/js/das_maps_dbs_prod.js, and it should contain JSON records. It is safe to delete the fie as it should be re-downloaded again. And, as I wrote before, all maps should be corrected to use cmsweb-auth.cern.ch instead of cmsweb.cern.ch:8443. For your convenience I put corrected maps here /afs/cern.ch/user/v/valya/public/das_maps_dbs_prod.js, so you can use this file instead. Just copy it to your $HOME/.dasmaps area.

oshadura commented 3 years ago

@vkuznet Ah now it makes sense! In my case, file which was pregenerated and in both cases: locally and at AF, had just inside next line:

cat $HOME/.dasmaps/das_maps_dbs_prod.js                                                     12:42:18 
404: Not Found%

which I replaced with 'cmsweb-auth.cern.ch' -> and of course it didn't work...

vkuznet commented 3 years ago

so, I"m confused now, did you try with proper das map file, does it work in this case? I see increasing number of failures from github these days and I should account for that in a future releases (may be we'll put dasmaps somewhere on cvmfs).

oshadura commented 3 years ago

Sorry for confusion, locally I am getting an empty result, but I think because https://cmsweb-auth.cern.ch/ is not available outside of CERN network? see https://gist.github.com/oshadura/8d1c2d8f2c86b5c979d1460ef1cf2994

From AF, I can't check now because it is in downtime this afternoon, I will let you know when it will be back.

vkuznet commented 3 years ago

Oksana, did you get time to test new version? Since I didn't hear from you I went ahead and included changes to new PR which is now passed all tests and merged. It means that new version of dasgoclient on cvmfs will have this feature. The new client is available on

/cvmfs/cms.cern.ch/common/dasgoclient -version
Build: git=v02.04.34 go=go1.16.3 date=2021-04-16 15:02:43.01225129 +0200 CEST m=+0.021729110

oshadura commented 3 years ago

@vkuznet sorry for such a delay! we had some work on progress on facility and I couldn't test properly dasgoclient:

cms-jovyan@jupyter-oksana-2eshadura-40cern-2ech:~/dasgoclient$ ./dasgoclient_amd64 -query "dataset=/ZMM*/*/*" -json -host https://cmsweb-auth.cern.ch -token /etc/cmsaf-secrets/xcache_token -verbose 3
DBSUrl:  https://cmsweb.cern.ch:8443/dbs/prod/DBSReader
SitedbUrl:  https://cmsweb.cern.ch:8443/sitedb/data/prod
CricUrl w/ site API:  https://cms-cric.cern.ch/api/cms/site/query
RucioUrl:  http://cms-rucio.cern.ch
RucioAuthUrl:  https://cms-rucio-auth.cern.ch/auth/x509
Load dasmaps /home/cms-jovyan/.dasmaps/das_maps_dbs_prod.js
2021/04/22 09:40:03 process, idx 0, val dataset, nval =, nnval /ZMM*/*/*

dataset=/ZMM*/*/*

### unique true
DAS map lookup, system dbs3, urn dataset4block, lookup [dataset], required keys [block], all keys [block]
DAS map lookup, system dbs3, urn dataset4parent_release, lookup [dataset], required keys [parent release], all keys [parent release]
DAS map lookup, system dbs3, urn datasets, lookup [dataset], required keys [], all keys [dataset primary_dataset datatype tier release run file era group status date user prepid]
DAS match: system=dbs3 urn=datasets url=https://cmsweb-auth.cern.ch/dbs/int/global/DBSReader/datasets/ spec keys=[dataset] requested keys=[] all api keys [dataset primary_dataset datatype tier release run file era group status date user prepid]
DAS map lookup, system dbs3, urn datasetlist, lookup [dataset], required keys [dataset], all keys [dataset status]
DAS match: system=dbs3 urn=datasetlist url=local_api spec keys=[dataset] requested keys=[dataset] all api keys [dataset status]
DAS map lookup, system mcm, urn dataset4mcm, lookup [dataset], required keys [prepid], all keys [prepid]
DAS map lookup, system phedex, urn blockReplicas, lookup [block], required keys [], all keys [block dataset site]
DAS map lookup, system phedex, urn dataset4site, lookup [dataset], required keys [site], all keys [dataset site]
DAS map lookup, system phedex, urn dataset4site_group, lookup [dataset], required keys [site group], all keys [dataset site group]
DAS map lookup, system phedex, urn dataset4se, lookup [dataset], required keys [site], all keys [dataset site]
DAS map lookup, system phedex, urn dataset4se_group, lookup [dataset], required keys [site], all keys [dataset site group]
DAS map lookup, system phedex, urn site4dataset, lookup [site], required keys [dataset], all keys [dataset]
DAS map lookup, system phedex, urn fileReplicas4dataset, lookup [file], required keys [dataset], all keys [dataset site]
DAS map lookup, system rucio, urn dataset4site, lookup [dataset], required keys [], all keys [site]
DAS map lookup, system rucio, urn block4dataset, lookup [block], required keys [dataset], all keys [dataset]
DAS map lookup, system rucio, urn rules4dataset, lookup [rules], required keys [dataset], all keys [dataset]
DAS map lookup, system rucio, urn file4dataset, lookup [file], required keys [dataset], all keys [dataset]
DAS map lookup, system rucio, urn file4dataset_site, lookup [file], required keys [dataset site], all keys [dataset site]
DAS map lookup, system rucio, urn block4dataset_site, lookup [block], required keys [dataset site], all keys [dataset site]
### selected services [dbs3:datasets dbs3:datasetlist] [dataset.name dataset.name]
### selected urls map[https://cmsweb-auth.cern.ch/dbs/int/global/DBSReader/datasets/?dataset=%2FZMM%2A%2F%2A%2F%2A&dataset_access_type=VALID&detail=True:]
### selected localApis [map[das_map:[map[api_arg:dataset das_key:dataset pattern:\["/[\w-]+/[\w-]+/[A-Z-]+"(\,\s*"/[\w-]+/[\w-]+/[A-Z-]+)+"\] rec_key:dataset.name] map[api_arg:primary_ds_name das_key:primary_dataset rec_key:primary_dataset.name] map[api_arg:primary_ds_type das_key:datatype rec_key:datatype.name] map[api_arg:data_tier_name das_key:tier pattern:.*[A-Z].* rec_key:tier.name] map[api_arg:release_version das_key:release rec_key:release.name] map[api_arg:run_num das_key:run rec_key:run.run_number] map[api_arg:logical_file_name das_key:file rec_key:file.name] map[api_arg:acquisition_era_name das_key:era rec_key:era] map[api_arg:physics_group_name das_key:group rec_key:group.name] map[api_arg:dataset_access_type das_key:status rec_key:status.name] map[api_arg:cdate das_key:date rec_key:date] map[api_arg:create_by das_key:user rec_key:user.name] map[api_arg:prep_id das_key:prepid rec_key:prepid]] expire:900 format:JSON hash:68298a736534dffba8b219cd3b8d25bb instances:[prod/global prod/phys01 prod/phys02 prod/phys03 prod/caf int/global int/phys01 int/phys02 int/phys03 dev/global dev/phys01 dev/phys02 dev/phys03] lookup:dataset params:map[dataset:required dataset_access_type:VALID detail:True] system:dbs3 ts:1.605993907e+09 type:service url:local_api urn:datasetlist]]
2021/04/22 09:40:03 http request, UrlQueueSize 1, UrlQueueLimit 100
2021/04/22 09:40:03 http request &{Method:GET URL:https://cmsweb-auth.cern.ch/dbs/int/global/DBSReader/datasets/?dataset=%2FZMM%2A%2F%2A%2F%2A&dataset_access_type=VALID&detail=True Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept-Encoding:[identity] Authorization:[Bearer MDAxY2xvY2F0aW9uIFQyX1VTX05lYnJhc2thCjAwMzRpZGVudGlmaWVyIGI3NGVjMGRhLTQyYWMtNGQ5MC04OGRiLWIwMzNlNjcxZGExMwowMDE4Y2lkIG5hbWU6Y21zLWpvdnlhbgowMDFhY2lkIGFjdGl2aXR5OkRPV05MT0FECjAwMTRjaWQgcGF0aDovc3RvcmUKMDAyNGNpZCBiZWZvcmU6MjAyMi0wNC0xOFQxNDoyODoyN1oKMDAyZnNpZ25hdHVyZSCEuVIYWLm00aWwhbRvDl2kf6_jPn285lqUGkZYZfY0wgo] User-Agent:[dasgoclient/v02.04.34]] Body:{} GetBody:<nil> ContentLength:0 TransferEncoding:[] Close:false Host:cmsweb-auth.cern.ch Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr: RequestURI: TLS:<nil> Cancel:<nil> Response:<nil> ctx:0xc00001a148}, rurl https://cmsweb-auth.cern.ch/dbs/int/global/DBSReader/datasets/?dataset=%2FZMM%2A%2F%2A%2F%2A&dataset_access_type=VALID&detail=True, dump GET /dbs/int/global/DBSReader/datasets/?dataset=%2FZMM%2A%2F%2A%2F%2A&dataset_access_type=VALID&detail=True HTTP/1.1
Host: cmsweb-auth.cern.ch
User-Agent: dasgoclient/v02.04.34
Accept-Encoding: identity
Authorization: Bearer 
XXXXXXXXXXXXXXX

, error <nil>

#### processURLs 1
DAS local API dbs3_datasetlist
### LOCAL APIS datasetlist dbs3 0 map[das_map:[map[api_arg:dataset das_key:dataset pattern:\["/[\w-]+/[\w-]+/[A-Z-]+"(\,\s*"/[\w-]+/[\w-]+/[A-Z-]+)+"\] rec_key:dataset.name] map[api_arg:primary_ds_name das_key:primary_dataset rec_key:primary_dataset.name] map[api_arg:primary_ds_type das_key:datatype rec_key:datatype.name] map[api_arg:data_tier_name das_key:tier pattern:.*[A-Z].* rec_key:tier.name] map[api_arg:release_version das_key:release rec_key:release.name] map[api_arg:run_num das_key:run rec_key:run.run_number] map[api_arg:logical_file_name das_key:file rec_key:file.name] map[api_arg:acquisition_era_name das_key:era rec_key:era] map[api_arg:physics_group_name das_key:group rec_key:group.name] map[api_arg:dataset_access_type das_key:status rec_key:status.name] map[api_arg:cdate das_key:date rec_key:date] map[api_arg:create_by das_key:user rec_key:user.name] map[api_arg:prep_id das_key:prepid rec_key:prepid]] expire:900 format:JSON hash:68298a736534dffba8b219cd3b8d25bb instances:[prod/global prod/phys01 prod/phys02 prod/phys03 prod/caf int/global int/phys01 int/phys02 int/phys03 dev/global dev/phys01 dev/phys02 dev/phys03] lookup:dataset params:map[dataset:required dataset_access_type:VALID detail:True] system:dbs3 ts:1.605993907e+09 type:service url:local_api urn:datasetlist] dbs3_datasetlist 0x4ab0e0 0
#### processLocalApis 1
Received 1 records
[
{"das":{"expire":1619084433,"instance":"prod/global","primary_key":"dataset.name","record":1,"services":["dbs3:datasets"]},"dataset":[{"das":{"expire":1619085033,"instance":"prod/global","primary_key":"dataset.name","record":0,"services":["das:NA"],"status":"ok","ts":1619084433},"name":"/ZMM*/*/*","qhash":"a23301d90474354ea00f8dd44c62c6b2","query":"dataset=/ZMM*/*/*"}],"qhash":"a23301d90474354ea00f8dd44c62c6b2"}
]

vkuznet commented 3 years ago

Oksana, you re-opened the ticket, is there anything else needs to be done here?

oshadura commented 3 years ago

Sorry for disturbing again, I actually accidentally closed it. I am not sure if it works on AF? I tried the example from README:

cms-jovyan@jupyter-oksana-2eshadura-40cern-2ech:~/dasgoclient$ ./dasgoclient_amd64 -query="file dataset=/ZMM/Summer11-DESIGN42_V11_428_SLHC1-v1/GEN-SIM" -host https://cmsweb-auth.cern.ch -token /etc/cmsaf-secrets/xcache_token
Fail to parse DAS record=map[das:map[expire:1619092927 instance:prod/global primary_key:file.name record:1 services:[dbs3:files_via_dataset]] file:[map[das:map[expire:1619093527 instance:prod/global primary_key:file.name record:0 services:[das:NA] status:ok ts:1619092927] qhash:f28499f53016567764d7ba76a3d76e4a query:file dataset=/ZMM/Summer11-DESIGN42_V11_428_SLHC1-v1/GEN-SIM]] qhash:f28499f53016567764d7ba76a3d76e4a], keys=[file [0] name], error=Key path not found

oshadura commented 3 years ago

Could it be that the next lines are the reason of a problem? (I reran the previous query with -verbose 3 )

....
2021/04/22 14:25:04 ERROR: unable to perform request Get "https://cms-rucio-auth.cern.ch/auth/x509": x509: certificate signed by unknown authority
rucio token  Get "https://cms-rucio-auth.cern.ch/auth/x509": x509: certificate signed by unknown authority
### selected services [dbs3:files_via_dataset] [file.name]
...

vkuznet commented 3 years ago

Oksana, here is on lxplus:

# I get token from https://cmsweb-auth.cern.ch/token

# I put token into /tmp/valya/token

# I setup DBS_URL
export DBS_URL=https://cmsweb-auth.cern.ch/dbs/int/global/DBSReader

# I use my map file which contains proper maps:
dasgoclient -query="dataset=/ZMM/Summer11-DESIGN42_V11_428_SLHC1-v1/GEN-SIM" -host https://cmsweb-auth.cern.ch -token /tmp/valya/token -dasmaps /afs/cern.ch/user/v/valya/public/das_maps_dbs_int.js

/ZMM/Summer11-DESIGN42_V11_428_SLHC1-v1/GEN-SIM

From my side everything works. Please try out this recipe.

vkuznet commented 3 years ago

Once again, since it is dev setup I need to set DBS_URL and use specific map file. Once we move to production these steps will not be necessary.

vkuznet commented 3 years ago

and regarding file query, yes it seems to me it is a problem. Rucio so far requires x509 and we need to work with them to get token based access. I need to ask Eric about it.

oshadura commented 3 years ago

Thanks a lot for your reply and sorry again for asking too many questions! Now I see it works from lxplus, but not locally from my laptop connected to the CERN network (a problem with x509: certificate signed by unknown authority) and from AF (here I am getting i/o timeout error).

oshadura commented 3 years ago

@bbockelm I think I will need to wait some time when it will be supported in prod...

vkuznet commented 3 years ago

the cmsweb-auth only visible at CERN network, so if your AF (analysis facility, right), is outside CERN then it will not work. And, I'll clear up details with Rucio team about x509 error.

vkuznet commented 3 years ago

@oshadura , @bbockelm could you please provide me instructions how should I get proper token with oidc-agent. I read the documentation and I have few questions:

when I run it I need to specify oidc-agent -w device <CLIENT_NAME>. What is CLIENT_NAME?
how I can get token for cmsweb-auth.cern.ch? What is a CLIENT_NAME in this case? Where should I enter cmsweb-auth.cern.ch?
I contacted Eric and he said that the token generated with oidc-agent should work with Rucio, therefore I need to generate token which will work for multiple services, e.g. cmsweb-auth and cms-rucio, etc. (later I may need token for other services). How I can generate token which can be used across all services?
- I already checked that if I get the token from cmsweb-auth CERN SSO I can't use it in Rucio, so I bet we either need to generate multiple tokens, one for each service, or have token authority to provide a multi-service token. I have no idea if it is possible.

What I tried is the following:

oidc-gen -w device cmsweb
...
Issuer [https://wlcg.cloud.cnaf.infn.it/]: 1
Scopes or 'max' (space separated) [openid profile offline_access]: openid profile email offline_access
Registering Client ...
Generating account configuration ...
accepted

Using a browser on another device, visit:
https://wlcg.cloud.cnaf.infn.it/device
...

oidc-token cmsweb

and with that token I CAN'T access cmsweb-auth. So I bet I need to properly specify client. The token I worked with are from cmsweb-auth itself since I never used WLCG INDIGO IAM for token generation before.

If I use

oidc-gen -w device cmsweb-auth
...
Issuer [https://wlcg.cloud.cnaf.infn.it/]: https://cmsweb-auth.cern.ch
Error while retrieving supported scopes for 'https://cmsweb-auth.cern.ch/': Could not get token endpoint from the configuration endpoint. This could be because of a network issue. But it's more likely that your issuer is not correct.

So there is no cmsweb-auth issuer. Then the question is how we manage issuers? How we can translate cmsweb-auth CERN SSO into WLGC IAM? So many questions and so little documentation. It would be nice if the documentation will be improved to cover all issues I outlined here.

Please note, that I setup cmsweb-auth.cern.ch in complains with CERN SSO application portal

clundst commented 1 year ago

I was trying to go through the recipe on lxplus and I'm still having issues with tokens and dasgoclient:

./dasgoclient_amd64 --version Build: git=v02.04.33 go=go1.16.3 date=2023-04-20 22:16:23.550027141 +0200 CEST m=+0.069117607

It's not clear to me what's going wrong here:

[clundst@lxplus702 ~]$ ./dasgoclient_amd64 -query "dataset=/ZMM*/*/*" -json -host https://cmsweb-auth.cern.ch -token token -verbose 3
DBSUrl:  https://cmsweb.cern.ch:8443/dbs/prod/DBSReader
SitedbUrl:  https://cmsweb.cern.ch:8443/sitedb/data/prod
CricUrl w/ site API:  https://cms-cric.cern.ch/api/cms/site/query
RucioUrl:  http://cms-rucio.cern.ch
RucioAuthUrl:  https://cms-rucio-auth.cern.ch/auth/x509
Load dasmaps /afs/cern.ch/user/c/clundst/.dasmaps/das_maps_dbs_prod.js
2023/04/20 22:09:26 process, idx 0, val dataset, nval =, nnval /ZMM*/*/*

dataset=/ZMM*/*/*

### unique true
DAS map lookup, system dbs3, urn dataset4block, lookup [dataset], required keys [block], all keys [block]
DAS map lookup, system dbs3, urn dataset4parent_release, lookup [dataset], required keys [parent release], all keys [parent release]
DAS map lookup, system dbs3, urn datasets, lookup [dataset], required keys [], all keys [dataset primary_dataset datatype tier release run file era group status date user prepid]
DAS match: system=dbs3 urn=datasets url=https://cmsweb-auth.cern.ch/dbs/int/global/DBSReader/datasets/ spec keys=[dataset] requested keys=[] all api keys [dataset primary_dataset datatype tier release run file era group status date user prepid]
DAS map lookup, system dbs3, urn datasetlist, lookup [dataset], required keys [dataset], all keys [dataset status]
DAS match: system=dbs3 urn=datasetlist url=local_api spec keys=[dataset] requested keys=[dataset] all api keys [dataset status]
DAS map lookup, system mcm, urn dataset4mcm, lookup [dataset], required keys [prepid], all keys [prepid]
DAS map lookup, system phedex, urn blockReplicas, lookup [block], required keys [], all keys [block dataset site]
DAS map lookup, system phedex, urn dataset4site, lookup [dataset], required keys [site], all keys [dataset site]
DAS map lookup, system phedex, urn dataset4site_group, lookup [dataset], required keys [site group], all keys [dataset site group]
DAS map lookup, system phedex, urn dataset4se, lookup [dataset], required keys [site], all keys [dataset site]
DAS map lookup, system phedex, urn dataset4se_group, lookup [dataset], required keys [site], all keys [dataset site group]
DAS map lookup, system phedex, urn site4dataset, lookup [site], required keys [dataset], all keys [dataset]
DAS map lookup, system phedex, urn fileReplicas4dataset, lookup [file], required keys [dataset], all keys [dataset site]
DAS map lookup, system rucio, urn dataset4site, lookup [dataset], required keys [], all keys [site]
DAS map lookup, system rucio, urn block4dataset, lookup [block], required keys [dataset], all keys [dataset]
DAS map lookup, system rucio, urn rules4dataset, lookup [rules], required keys [dataset], all keys [dataset]
DAS map lookup, system rucio, urn file4dataset, lookup [file], required keys [dataset], all keys [dataset]
DAS map lookup, system rucio, urn file4dataset_site, lookup [file], required keys [dataset site], all keys [dataset site]
DAS map lookup, system rucio, urn block4dataset_site, lookup [block], required keys [dataset site], all keys [dataset site]
### selected services [dbs3:datasets dbs3:datasetlist] [dataset.name dataset.name]
### selected urls map[https://cmsweb-auth.cern.ch/dbs/int/global/DBSReader/datasets/?dataset=%2FZMM%2A%2F%2A%2F%2A&dataset_access_type=VALID&detail=True:]
### selected localApis [map[das_map:[map[api_arg:dataset das_key:dataset pattern:\["/[\w-]+/[\w-]+/[A-Z-]+"(\,\s*"/[\w-]+/[\w-]+/[A-Z-]+)+"\] rec_key:dataset.name] map[api_arg:primary_ds_name das_key:primary_dataset rec_key:primary_dataset.name] map[api_arg:primary_ds_type das_key:datatype rec_key:datatype.name] map[api_arg:data_tier_name das_key:tier pattern:.*[A-Z].* rec_key:tier.name] map[api_arg:release_version das_key:release rec_key:release.name] map[api_arg:run_num das_key:run rec_key:run.run_number] map[api_arg:logical_file_name das_key:file rec_key:file.name] map[api_arg:acquisition_era_name das_key:era rec_key:era] map[api_arg:physics_group_name das_key:group rec_key:group.name] map[api_arg:dataset_access_type das_key:status rec_key:status.name] map[api_arg:cdate das_key:date rec_key:date] map[api_arg:create_by das_key:user rec_key:user.name] map[api_arg:prep_id das_key:prepid rec_key:prepid]] expire:900 format:JSON hash:68298a736534dffba8b219cd3b8d25bb instances:[prod/global prod/phys01 prod/phys02 prod/phys03 prod/caf int/global int/phys01 int/phys02 int/phys03 dev/global dev/phys01 dev/phys02 dev/phys03] lookup:dataset params:map[dataset:required dataset_access_type:VALID detail:True] system:dbs3 ts:1.605993907e+09 type:service url:local_api urn:datasetlist]]
2023/04/20 22:09:26 http request, UrlQueueSize 1, UrlQueueLimit 100
2023/04/20 22:09:26 http request &{Method:GET URL:https://cmsweb-auth.cern.ch/dbs/int/global/DBSReader/datasets/?dataset=%2FZMM%2A%2F%2A%2F%2A&dataset_access_type=VALID&detail=True Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Accept-Encoding:[identity] Authorization:[Bearer ....] User-Agent:[dasgoclient/v02.04.33]] Body:{} GetBody:<nil> ContentLength:0 TransferEncoding:[] Close:false Host:cmsweb-auth.cern.ch Form:map[] PostForm:map[] MultipartForm:<nil> Trailer:map[] RemoteAddr: RequestURI: TLS:<nil> Cancel:<nil> Response:<nil> ctx:0xc000016148}, rurl https://cmsweb-auth.cern.ch/dbs/int/global/DBSReader/datasets/?dataset=%2FZMM%2A%2F%2A%2F%2A&dataset_access_type=VALID&detail=True, dump GET /dbs/int/global/DBSReader/datasets/?dataset=%2FZMM%2A%2F%2A%2F%2A&dataset_access_type=VALID&detail=True HTTP/1.1
Host: cmsweb-auth.cern.ch
User-Agent: dasgoclient/v02.04.33
Accept-Encoding: identity
Authorization: Bearer ......

, error <nil>
#### processURLs 1
DAS local API dbs3_datasetlist
### LOCAL APIS datasetlist dbs3 0 map[das_map:[map[api_arg:dataset das_key:dataset pattern:\["/[\w-]+/[\w-]+/[A-Z-]+"(\,\s*"/[\w-]+/[\w-]+/[A-Z-]+)+"\] rec_key:dataset.name] map[api_arg:primary_ds_name das_key:primary_dataset rec_key:primary_dataset.name] map[api_arg:primary_ds_type das_key:datatype rec_key:datatype.name] map[api_arg:data_tier_name das_key:tier pattern:.*[A-Z].* rec_key:tier.name] map[api_arg:release_version das_key:release rec_key:release.name] map[api_arg:run_num das_key:run rec_key:run.run_number] map[api_arg:logical_file_name das_key:file rec_key:file.name] map[api_arg:acquisition_era_name das_key:era rec_key:era] map[api_arg:physics_group_name das_key:group rec_key:group.name] map[api_arg:dataset_access_type das_key:status rec_key:status.name] map[api_arg:cdate das_key:date rec_key:date] map[api_arg:create_by das_key:user rec_key:user.name] map[api_arg:prep_id das_key:prepid rec_key:prepid]] expire:900 format:JSON hash:68298a736534dffba8b219cd3b8d25bb instances:[prod/global prod/phys01 prod/phys02 prod/phys03 prod/caf int/global int/phys01 int/phys02 int/phys03 dev/global dev/phys01 dev/phys02 dev/phys03] lookup:dataset params:map[dataset:required dataset_access_type:VALID detail:True] system:dbs3 ts:1.605993907e+09 type:service url:local_api urn:datasetlist] dbs3_datasetlist 0x4ab0e0 0
#### processLocalApis 1
Received 1 records
[
{"das":{"expire":1682021366,"instance":"prod/global","primary_key":"dataset.name","record":1,"services":["dbs3:datasets"]},"dataset":[{"das":{"expire":1682021966,"instance":"prod/global","primary_key":"dataset.name","record":0,"services":["das:NA"],"status":"ok","ts":1682021366},"name":"/ZMM*/*/*","qhash":"a23301d90474354ea00f8dd44c62c6b2","query":"dataset=/ZMM*/*/*"}],"qhash":"a23301d90474354ea00f8dd44c62c6b2"}

]

vkuznet commented 1 year ago

@clundst , nothing is wrong with DAS per-se. If you'll check its output you'll see that DAS redirects your query to DBS instance at this URL: https://cmsweb-auth.cern.ch/dbs/int/global/DBSReader/datasets/?dataset=%2FZMM%2A%2F%2A%2F%2A&dataset_access_type=VALID&detail=True, and we do not run DBS server on cmsweb-auth server and therefore DAS gets nothing from it.

My time is no longer allocated for DAS/CMSWEB/tokens, and I can't help you with that. Please ask @klannon about additional man-power to resolve this issue. The migration to token based authentication so far is suspended on CMSWEB as far as I know and no one maintain cmsweb-auth cluster.

dmwm / dasgoclient

Add possibility to use bearer tokens instead of X509 proxy #30