OIDC: dCacheView has client scope hardcoded, ignores client definition in IAM OIDC provider

[onnozweers]

Dear dCache devs,

I couldn't get dCacheView on our 9.1 test instance to authenticate with group memberships. You may have seen my posts about this on the dCache user mailing list. In my client definition at the Escape IAM (https://iam-escape.cloud.cnaf.infn.it/iam/api/client-registration/aa4d4818-0e88-4f47-90b9-dd2ef5e84cf8) I added wlcg.groups to the scope, but it was always ignored; in the token that was returned to gPlazma, the scope was always this:

"scope": "openid profile email"

I just found out that this is hardcoded in the dCacheView source:

https://github.com/dCache/dcache-view/blob/6afb479fe749bfb4d15fb288345e911d61d256d4/src/elements/dv-elements/user-authentication/forms/loginform-with-openid.html#L297

Any suggestions on how to configure user mappings based on wlcg.groups information? Or is this not supported in dCacheView, and should we look for another client to test OIDC? For now I guess we could add wlcg.groups to the source, but it feels a bit clunky.

Kind regards, Onno

[onnozweers]

I tried to work around it:

[root@hedgehog14 /usr/share/dcache/dcache-view]# grep -R -3 scope.*openid *
elements/elements.html-                window.location.href = endpoint + 'response_type=id_token%20token' +
elements/elements.html-                    '&client_id=' + openIDprovider.id +
elements/elements.html-                    '&redirect_uri=' + encodeURIComponent(window.location.origin) +
elements/elements.html:                    '&scope=openid%20profile%20email%20wlcg.groups' +
elements/elements.html-                    '&nonce=' + nonce + extra;
elements/elements.html-            }
elements/elements.html-

But that doesn't work. I wasn't able to find any other location where this scope is defined. Clearly my understanding of dCacheView and how it is integrated into dCache is insufficient to change this myself.

[paulmillar]

The change looks reasonable to me: I can't see anything obviously wrong.

It could be that your client needs to be authorised to "see" (i.e., to obtain a token with) the wlcg.groups claim. That authorisation would be something configurable within IAM, perhaps requiring admin intervention.

[onnozweers]

Hi Paul,

I have checked the client definition over and over again, but the wlcg.groups is checked in the scope. So I really don't understand why it is not included in the token.

The token payload:

{
  "wlcg.ver": "1.0",
  "sub": "c7f64da7-15e7-4985-afb8-3c8ec96dbb80",
  "aud": "https://wlcg.cern.ch/jwt/v1/any",
  "nbf": 1683723795,
  "scope": "openid profile email",
  "iss": "https://iam-escape.cloud.cnaf.infn.it/",
  "exp": 1683727395,
  "iat": 1683723795,
  "jti": "6b0f3e33-33ad-4903-bd75-b1341edeaeb6",
  "client_id": "aa4d4818-0e88-4f47-90b9-dd2ef5e84cf8"
}

The client ID is correctly defined, I hope:

[root@hedgehog14 /etc/dcache]# grep '^[^#]'.*oidc layouts/hedgehog14.conf 
gplazma.oidc.provider!ESCAPE=https://iam-escape.cloud.cnaf.infn.it/ -profile=wlcg -prefix=/ -suppress=offline
gplazma.oidc.audience-targets = https://wlcg.cern.ch/jwt/v1/any
frontend.static!dcache-view.oidc-provider-name-list = ESCAPE
frontend.static!dcache-view.oidc-client-id-list = aa4d4818-0e88-4f47-90b9-dd2ef5e84cf8
frontend.static!dcache-view.oidc-authz-endpoint-list = https://iam-escape.cloud.cnaf.infn.it/authorize
frontend.static!dcache-view.oidc-authz-endpoint-extra = -

I have tried debugging the IAM traffic with the browser console, but somehow I couldn't see that traffic; only the dCacheView-browser traffic.

[onnozweers]

Now here's a weird thing. My colleague Natalie just registered at the Escape IAM, and got her ID (sub). When she gets a token, the wlcg.groups are included in the scope:

{
  "wlcg.ver": "1.0",
  "sub": "10768676-9112-4ea5-b2db-c3aca5003b1f",
  "aud": "https://wlcg.cern.ch/jwt/v1/any",
  "nbf": 1683724514,
  "scope": "openid profile email wlcg.groups",
  "iss": "https://iam-escape.cloud.cnaf.infn.it/",
  "exp": 1683728114,
  "iat": 1683724514,
  "jti": "47eaf3f6-0170-4af2-9bde-e487b2a1b4e1",
  "client_id": "aa4d4818-0e88-4f47-90b9-dd2ef5e84cf8"
}

She's not in any groups yet, though. That's probably why there are no groups listed.

[onnozweers]

For Natalie, after she was admitted to the groups, we now receive the group info in dCache.

{
  "wlcg.ver": "1.0",
  "sub": "10768676-9112-4ea5-b2db-c3aca5003b1f",
  "aud": "https://wlcg.cern.ch/jwt/v1/any",
  "nbf": 1683801434,
  "scope": "openid profile email wlcg.groups",
  "iss": "https://iam-escape.cloud.cnaf.infn.it/",
  "exp": 1683805034,
  "iat": 1683801434,
  "jti": "11630268-25b5-42b7-bc77-3321c7024297",
  "client_id": "aa4d4818-0e88-4f47-90b9-dd2ef5e84cf8",
  "wlcg.groups": [
    "/escape",
    "/escape/ska"
  ]
}

For me, we still don't. Here's a fresh token payload:

{
  "wlcg.ver": "1.0",
  "sub": "c7f64da7-15e7-4985-afb8-3c8ec96dbb80",
  "aud": "https://wlcg.cern.ch/jwt/v1/any",
  "nbf": 1683804876,
  "scope": "openid profile email",
  "iss": "https://iam-escape.cloud.cnaf.infn.it/",
  "exp": 1683808476,
  "iat": 1683804876,
  "jti": "f0a951d1-87bb-46b5-948d-f6aad1d2d59e",
  "client_id": "aa4d4818-0e88-4f47-90b9-dd2ef5e84cf8"
}

Does dCache somehow cache things like the scope in a user profile?

Authentication still fails, also for Natalie, because of this error:

 |   +--oidc OPTIONAL:FAIL (wlcg.ver claim is missing) => OK
 |          added: OP[ESCAPE]

I really have no idea where that came from. Haven't seen it before yesterday.

[paulmillar]

The wlcg.ver claim is missing message comes from the WLCG profile. This requires that the token has a wlcg.ver claim.

It looks like this claim was somehow missing.

[onnozweers]

But Paul, I can see the "wlcg.ver": "1.0" in the tokens. So, how can it be considered missing? Or is it missing from some location I'm not aware of?

I'm using yesterday's master snapshot.

[paulmillar]

There's (at least) three possibilities:

1. the token dCache sees isn't the token you're seeing
2. the token is sent to the user-info endpoint, which is returning a different set of claims
3. there a bug in dCache code.

[onnozweers]

1. The token payloads I pasted above, I took from the "Found bearer token" line in the gPlazma log at debug level, decoded by jwt.io.
2. Not sure what that means.

Happy to help debugging this, tell me what you need.

[paulmillar]

Here's how you can look up a token, using the user-info endpoint. I'm using oidc-agent to get the token (as TOKEN) in this example, but you would use the token from dCache/gPlazma.

paul@celebrimbor:~$ curl -s https://iam-escape.cloud.cnaf.infn.it/.well-known/openid-configuration | jq -r .userinfo_endpoint
https://iam-escape.cloud.cnaf.infn.it/userinfo
paul@celebrimbor:~$ TOKEN=$(oidc-token ESCAPE)
paul@celebrimbor:~$ curl -s -H "Authorization: Bearer $TOKEN" https://iam-escape.cloud.cnaf.infn.it/userinfo | jq .
{
  "sub": "c95caedb-97dd-4335-bbb9-b31919961132",
  "name": "Paul Millar",
  "preferred_username": "paul",
  "given_name": "Paul",
  "family_name": "Millar",
  "updated_at": 1605380764
}
paul@celebrimbor:~$ 

The first curl command is to look up the userinfo endpoint. I think you can just use the value https://iam-escape.cloud.cnaf.infn.it/userinfo without worrying about how to obtain this information.

[paulmillar]

In general, dCache has a couple of ways it can obtain information about the user from a token. If the token is a JWT then dCache can obtain the information directly; otherwise, it can contact the user-info endpoint.

One possible explanation is that dCache is (for whatever reason) using the user-info endpoint with your token, instead of reading the data directly from the JWT.

[onnozweers]

Thanks Paul! Here's my user info:

onno$ curl -s -H "Authorization: Bearer $TOKEN" https://iam-escape.cloud.cnaf.infn.it/userinfo | jq .
{
  "sub": "c7f64da7-15e7-4985-afb8-3c8ec96dbb80",
  "name": "Onno Zweers",
  "preferred_username": "OnnoZweers",
  "given_name": "Onno",
  "family_name": "Zweers",
  "picture": "https://images-na.ssl-images-amazon.com/images/I/717bQJ0K3cL._SL1500_.jpg",
  "updated_at": 1683734777,
  "wlcg.groups": [
    "/escape",
    "/escape/ska"
  ]
}

That was with a fresh token from oidc-token.

Then I tried a token I obtained from the gPlazma log:

16:25 ui.grid.surfsara.nl:/home/onno 
onno$ curl -s -H "Authorization: Bearer $TOKEN" https://iam-escape.cloud.cnaf.infn.it/userinfo | jq .
{
  "sub": "c7f64da7-15e7-4985-afb8-3c8ec96dbb80",
  "name": "Onno Zweers",
  "preferred_username": "OnnoZweers",
  "given_name": "Onno",
  "family_name": "Zweers",
  "picture": "https://images-na.ssl-images-amazon.com/images/I/717bQJ0K3cL._SL1500_.jpg",
  "updated_at": 1683734777,
  "email": "onno.zweers@surf.nl",
  "email_verified": true
}

[onnozweers]

Indeed, in the gPlazma log I found that gPlazma reaches out to the /userinfo.

11 May 2023 16:24:41 (gPlazma) [] http-outgoing-9 >> "GET /userinfo HTTP/1.1[\r][\n]"
11 May 2023 16:24:41 (gPlazma) [] http-outgoing-9 >> "Authorization: Bearer **************
[\r][\n]"

The token used has this payload:

{
  "wlcg.ver": "1.0",
  "sub": "c7f64da7-15e7-4985-afb8-3c8ec96dbb80",
  "aud": "https://wlcg.cern.ch/jwt/v1/any",
  "nbf": 1683815078,
  "scope": "openid profile email",
  "iss": "https://iam-escape.cloud.cnaf.infn.it/",
  "exp": 1683818678,
  "iat": 1683815078,
  "jti": "3e0db043-070f-449c-b6c5-4d054ea82237",
  "client_id": "aa4d4818-0e88-4f47-90b9-dd2ef5e84cf8"
}

For Natalie, gPlazma does the same, but it uses a token with this payload:

{
  "wlcg.ver": "1.0",
  "sub": "10768676-9112-4ea5-b2db-c3aca5003b1f",
  "aud": "https://wlcg.cern.ch/jwt/v1/any",
  "nbf": 1683801434,
  "scope": "openid profile email wlcg.groups",
  "iss": "https://iam-escape.cloud.cnaf.infn.it/",
  "exp": 1683805034,
  "iat": 1683801434,
  "jti": "11630268-25b5-42b7-bc77-3321c7024297",
  "client_id": "aa4d4818-0e88-4f47-90b9-dd2ef5e84cf8",
  "wlcg.groups": [
    "/escape",
    "/escape/ska"
  ]
}

That probably explains why the /userinfo result is different.

I've collected some debug logs and sent them to your email address.

[onnozweers]

I made a small step forward, regarding the wlcg.ver claim is missing error.

It turns out, that when I remove the -suppress=offline from this line:

[root@hedgehog14 /etc/dcache]# grep offline layouts/hedgehog14.conf 
gplazma.oidc.provider!ESCAPE=https://iam-escape.cloud.cnaf.infn.it/ -profile=wlcg -prefix=/ -suppress=offline

and restart dCache (not sure if that is needed), the wlcg.ver error disappears! But to be honest, I have no idea why that is. No idea even if that is a "feature" of dCache or of the Escape IAM.

[paulmillar]

Actually, I think this is pretty clear.

Earlier you posted the JWT payload that identifies yourself. This included the wlcg.ver claim.

You also posted the user-info endpoint response, when you use your access token (i.e., the JWT) to authenticate. This clearly does NOT include the wlcg.ver claim.

You have configured dCache to expect the OP to conform to the WLCG JWT Profile. This profile requires the OP to include the wlcg.ver claim.

If you do not include -suppress=offline then dCache sees the JWT, does the offline verification and extracts information from the JWT payload. As you've already demonstrated, the JWT payload includes the wlcg.ver claim.

With -suppress=offline option enabled, dCache can no longer do the offline verification, so it falls back to using the information from the user-info endpoint (online verification). Again, as you've already demonstrated, the user-info endpoint does NOT include the wlcg.ver claim.

Therefore, the error message (wlcg.ver claim is missing) is consistent with observed behaviour of the OP and dCache's configuration. (At least, it makes sense to me!)

[onnozweers]

Thanks for explaining! I'm glad you understand. Looking forward to the workshop to improve my understanding of OIDC.

It's still not clear to me whether this is desired behaviour. Should the userinfo include wlcg.ver? Or should dCache accept that the userinfo doesn't have wlcg.ver?

[paulmillar]

I think it's rather unclear what is correct behaviour.

Switching off wlcg.ver would (in effect) switch off the WLCG/JWT profile within dCache. This is possible, but then we would loose the group-membership information (which was the whole point of trying -suppress=offline).

So, I would say the way forward would be to switch off -suppress=offline (it was a gambit, but one that didn't pay off). I would recommend focusing on getting the group-membership information included in the JWT's payload. This (very likely) requires configuration changes within the OP.