Google Groups claim working for some users but not for others

[Pionerd]

Current Behavior

We configured Google Groups validation, however this is working for some users, but not for others.

Non functioning user:

Mar 21 14:15:03 <hostname> openvpn-auth-oauth2[2010]: time=2024-03-21T14:15:03.407Z level=WARN msg="user validation: missing claim: groups" ip=<ip>:51235 cid=2 kid=1 session_id="" common_name="" idtoken.subject=<user_id> idtoken.email=<user_email> idtoken.preferred_username="" user.subject=<same user_id>  user.preferred_username="" error_id=<random>

Functioning user:

Mar 21 14:25:38 <hostname> openvpn-auth-oauth2[2010]: time=2024-03-21T14:25:38.911Z level=INFO msg="accept OpenVPN client cid 6, kid 1" ip=<ip>:51080 cid=6 kid=1 session_id="" common_name="" idtoken.subject=<user_id> idtoken.email=<user_email> idtoken.preferred_username="" user.subject=<user_id> user.preferred_username=""

Expected Behavior

Groups validation should work for all users

Steps To Reproduce

No response

Environment

• openvpn-auth-oauth2 Version: latest
• OpenVPN Server Version: 2.6.9-jammy0
• Server OS: Ubuntu
• OpenVPN Client (flavor, OS):
• OIDC Provider: Google

openvpn-auth-oauth2 logs

See above

openvpn server logs

N/A

Anything else?

The users for who this is not functioning are in the correct group. However, what may make this special, is that they are "external users" who are imported using a sync with Azure Active Directory (as opposed to the native accounts that are working). The format of the email addresses that are not working is name_CompanyA.com@CompanyB.com where CompanyB is the Google Organisation.

[jkroepke]

Which OIDC claim contains name_CompanyA.com@CompanyB.com?

[Pionerd]

I'm not sure what would be the best way to determine that, but in the browser I see this URL passing by:

https://<vpn_domain>:9000/oauth2/callback?state=<base64_code>&code=<another_random_string>&scope=email%20profile%20https://www.googleapis.com/auth/userinfo.email%20https://www.googleapis.com/auth/userinfo.profile%20openid%20https://www.googleapis.com/auth/cloud-identity.groups.readonly&authuser=0&hd=<CompanyB>.com&prompt=consent

[jkroepke]

I will add some debug log lines for that.

[jkroepke]

Try https://github.com/jkroepke/openvpn-auth-oauth2/releases/tag/v1.19.3 in debug mode and search for claims

[jkroepke]

I may expect, that this bug hits Grafana, too.

[Pionerd]

[jkroepke]

I checked multiple docs and the major issue here is that Cloud Identity API works with "Member Key", while this information seems not available via OIDC. Instead a "Member Key", which based on name_CompanyA.com@CompanyB.com, I only get the ID of the Google Account (numeric value).

[Pionerd]

Would the "old" way of doing things (using a Service Account) have the same issue?

[jkroepke]

No clue, but the Admin API works with ID of an Google Identity, instead an Member Key (mail) and the ID was present on the claim, too.

[jkroepke]

Could you please test the following steps?

1. Go to the Admin Portal of Google and go to the group that is required for openvpn-auth-oauth2.
2. The URL should be in the format of https://admin.google.com/ac/groups/<ID>. Copy the ID
3. Go to the Cloud identity API. https://cloud.google.com/identity/docs/reference/rest/v1/groups.memberships/list
4. On the left side, enter groups/<ID> as parent
5. Then, run Try-Out and check, if the API returns external API users as well.

Also please let me know, if the member key of the external user is present in token claims. (Using 1.20, run in debug mode and grep for claim)

In case it works, then I would adjust the docs, that the ID of group is mandatory. The chance is high, that is is also working with external synced groups.

[Pionerd]

I'm awaiting response on the API test, but I can see the following claims, nothing seems related to the member keys.

claims="map[
at_hash: <redacted>
aud: <redacted>
azp: <redacted>
email: <redacted>
email_verified:true
exp: <redacted>
family_name:_
given_name:_
hd: <redacted>
iat: <redacted>
iss:https://accounts.google.com
name:__
nonce: <redacted>
picture: <redacted>
sub: <redacted>

[Pionerd]

What was the latest version of your app with the old method for group validation?

[Pionerd]

I can confirm the external users do show up in the API call

[jkroepke]

What was the latest version of your app with the old method for group validation?

1.17.x

I can confirm the external users do show up in the API call

If you want, you can also try an experimental build, where oauth2.validate.groups require the mentioned Group ID, that you are used at the API call.

Binary: https://github.com/jkroepke/openvpn-auth-oauth2/actions/runs/8538208957/artifacts/1380990698

CONFIG_OAUTH2_VALIDATE_GROUPS=03x8tuzt3hqdv5v

[Pionerd]

Apr 03 15:56:21 shared-hub-vpn-gateway openvpn-auth-oauth2[11001]: time=2024-04-03T15:56:21.407Z level=WARN msg="user validation: error from Google API https://cloudidentity.googleapis.com/v1/groups/<GROUP_ID>/memberships: http status code: 403; message: Error(4001): Permission denied for membership resource 'groups/<GROUP_ID>' (or it may not exist)." ip=<ip>:54599 cid=0 kid=1 session_id="" common_name="" idtoken.subject=<redacted> idtoken.email=<redacted>  idtoken.preferred_username="" user.subject=<redacted>  user.preferred_username="" error_id=<redacted> 

[jkroepke]

Did you double check, that you put the ID (and only the ID) of the Group into CONFIG_OAUTH2_VALIDATE_GROUPS? It works at least with internal users on my personal Google Workspace.

[Pionerd]

I did put in only the ID of the group. I can check once more later for typos

Verzonden vanuit Outlook voor iOShttps://aka.ms/o0ukef

---

Van: Jan-Otto Kröpke @.> Verzonden: Wednesday, April 3, 2024 6:12:52 PM Aan: jkroepke/openvpn-auth-oauth2 @.> CC: Pieter van der Giessen @.>; Author @.> Onderwerp: Re: [jkroepke/openvpn-auth-oauth2] Google Groups claim working for some users but not for others (Issue #218)

Did you double check, that you put the ID (and only the ID) of the Group into CONFIG_OAUTH2_VALIDATE_GROUPS? It works at least with internal users on my Google Workspace.

— Reply to this email directly, view it on GitHubhttps://github.com/jkroepke/openvpn-auth-oauth2/issues/218#issuecomment-2035026890, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A2IQWGCN4T63GOJU4MVVYB3Y3QTAJAVCNFSM6AAAAABFBURNNSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZVGAZDMOBZGA. You are receiving this because you authored the thread.Message ID: @.***>

[jkroepke]

What you can do here is but grab the groups/<GROUP_ID> from the logs and put the into the parent field at https://cloud.google.com/identity/docs/reference/rest/v1/groups.memberships/list

[jkroepke]

Other take could to validate, if the works with an internal user, but not with an external user.

I expect, that the OIDC scope is set, but maybe there is an overall restriction that external user are not allowed for an lookup memberships of groups.

[jkroepke]

@Pionerd did you had a chance to test #242?

[Pionerd]

I did the same test again, with the same result. I'm the external user here, I will try to get in touch with someone who has an internal account and let you know if that works.

What you can do here is but grab the groups/ from the logs and put the into the parent field at https://cloud.google.com/identity/docs/reference/rest/v1/groups.memberships/list

We did this before and there the external users do show up correctly.

[jkroepke]

I will try to get in touch with someone who has an internal account and let you know if that works.

I can confirm, that it works with internal accounts, since I have access Google Workspace.

In case, you can confirm it works with an external users, I'm more than happy.

We did this before and there the external users do show up correctly.

Then I will merge #242 which should fix the issue. please mention that VALIDATE_GROUPS now needs the ID of the group, not the name.

[Pionerd]

@jkroepke sorry, apparently I was not clear.

What you can do here is but grab the groups/ from the logs and put the into the parent field at https://cloud.google.com/identity/docs/reference/rest/v1/groups.memberships/list When I do that, the external user shows up. However, with the new config, I still CANNOT login being an external user.

I have an appointment on Monday with an internal user to test. If it works for him, then we know our setup is OK and the external users are just an exception. If it also does not work for him, there is a difference between your and my setup. I don't think this should be closed already, as I have not yet been able to successfully login (as external user).

[jkroepke]

I will wait with the release, and in worst case I will revert #242

[Pionerd]

So, the weirdest of things happened. I went on to test this with an internal user, so I enabled the validate.groups config again and it worked for him. But also for me.

I have no idea why, I literally uncommented the same lines as before. Sorry for wasting time on this.

Edit: my colleague suggests this may have been in a bug in Google's API that has been fixed in the meantime.

[jkroepke]

To clarify this:

You tested this with the binary from #242 ? Where validate.groups is using Group ID now? And it works both for internal and external users?

[Pionerd]

I tested this with the SNAPSHOT binary you provided above and yes, using GroupID it now magically works for both internal and external users 🎉

[Pionerd]

If you release a new version, we will test (and implement) that straight away.

[jkroepke]

GroupID it now magically works

Background:

Before that, I was using https://cloud.google.com/identity/docs/reference/rest/v1/groups.memberships/searchDirectGroups which requires a MemberKey. For internal users, MemberKey is the email address, but for external users, its a different value and the value was not provided by Google.

If you really solve that issues for other solution, like Grafana, it may valuable to ask the Google Support, if they could provide the MemberKey via an Token Claim. That would fully close the gab between CloudIdentity API and OIDC Token/API.

I solved the issues, by to a reverse search. I'm using this API https://cloud.google.com/identity/docs/reference/rest/v1/groups.memberships/list now, which requires Group ID as input. It give me all users back that are member of a group. As part of the response, all user IDs of the group members are exposed and Google provide the ID of an user entity as sub claim inside the OIDC ID Token.

[Pionerd]

Why I used the word magically is because your SNAPSHOT release didn't work initially, but today (without any changes on our end) it started working. That's why we suspect a change in Google's API

[jkroepke]

If you release a new version, we will test (and implement) that straight away.

https://github.com/jkroepke/openvpn-auth-oauth2/releases/tag/v1.21.0

[Pionerd]

@jkroepke And the issue is back, we are completely lost here. I tested both the new version, but also the snapshot version no longer works. Do you see anything wrong with this config?

http:
  baseurl: "https://<domain>:9000"
  cert: "/etc/openvpn-auth-oauth2/fullchain.pem"
  key: "/etc/openvpn-auth-oauth2/privkey.pem"
  listen: ":9000"
  secret: "<secret>"
  tls: true
openvpn:
  addr: "unix:///run/openvpn/server.sock"
  password: "<secret>"
  common-name:
    environment-variable-name: username
oauth2:
  issuer: "https://accounts.google.com"
  client:
    id: "<client_id>"
    secret: "<client_secret>"
  validate:
    groups:
      - "<group_id1_without/groups>"
      - "<group_id2_without/groups>"   # <!-- We also tested with 1 group, no luck
  refresh:
    enabled: true
    expires: 8h0m0s
    use-session-id: true
    secret: "<secret>"
  provider: google
  authorize-params: "prompt=consent

Log:

Apr 15 13:31:59 <hostname> openvpn-auth-oauth2[27276]: time=2024-04-15T13:31:59.862Z level=INFO msg="new client connection" ip=<ip>:62346 cid=1 kid=1 common_name="" reason=CONNECT session_id=<session_id> session_state=Initial
Apr 15 13:31:59 <hostname> openvpn-auth-oauth2[27276]: time=2024-04-15T13:31:59.862Z level=INFO msg="start pending auth" ip=<ip>:62346 cid=1 kid=1 common_name="" reason=CONNECT session_id=<session_id> session_state=Initial
Apr 15 13:32:00 <hostname> openvpn-auth-oauth2[27276]: time=2024-04-15T13:32:00.258Z level=INFO msg="initialize authorization via oauth2" ip=<ip>:62346 cid=1 kid=1 common_name=""
Apr 15 13:32:01 <hostname> openvpn-auth-oauth2[27276]: time=2024-04-15T13:32:01.317Z level=INFO msg="client disconnected" ip=: cid=0 common_name="" reason=DISCONNECT session_id="" session_state=""
Apr 15 13:32:45 <hostname> openvpn-auth-oauth2[27276]: time=2024-04-15T13:32:45.905Z level=INFO msg="deny OpenVPN client cid 1, kid 1" ip=<ip>:62346 cid=1 kid=1 session_id=<session_id> common_name="" idtoken.subject=<subject_id> idtoken.email=<ext_email> idtoken.preferred_username="" user.subject=<subject_id> user.preferred_username=""
Apr 15 13:32:45 <hostname> openvpn-auth-oauth2[27276]: time=2024-04-15T13:32:45.907Z level=WARN msg="user validation: error from Google API https://cloudidentity.googleapis.com/v1/groups/<group_id_1>/memberships: http status code: 403; message: Error(4001): Permission denied for membership resource 'groups/<group_id_1>' (or it may not exist)." ip=<ip>:62346 cid=1 kid=1 session_id=<session_id> common_name="" idtoken.subject=<subject_id> idtoken.email=<ext_email> idtoken.preferred_username="" user.subject=<subject_id> user.preferred_username="" error_id=3223b62b7eea6fa526adc2ff66a3537f0fb75f253d050386cd455c21f27d6010
Apr 15 13:32:48 <hostname> openvpn-auth-oauth2[27276]: time=2024-04-15T13:32:48.976Z level=INFO msg="client disconnected" ip=: cid=1 common_name="" reason=DISCONNECT session_id="" session_state=""

Unrelated: logging shows only 1 of the 2 groups being checked. But as said, we tested with a single group also.

[jkroepke]

Unrelated: logging shows only 1 of the 2 groups being checked.

If there is an error on API level, it will be not proceed.

Does the error happens only, if multiple groups are configured?

If a single group is configured, is the external test user part of that (first) group?

[Pionerd]

We tested with 1 group of which the external user is part, it failed with the same error as above. I still cannot explain why this did work correctly this morning.

~Stupid question: are the memberships somehow cached? Is it possible that the internal user (who in our case is an admin) retrieved all memberships after which the external user could login?~

Other question: how about these rights on the API page, shouldn't they be granted to the Client used?

[Pionerd]

Just verified with the same internal user: works for him, does not work for me (external). I asked him to set the above scope for the Client.

[jkroepke]

Other question: how about these rights on the API page, shouldn't they be granted to the Client used?

No. They are part of OIDC Token flow and the scope are granted to the token of the logged-in user. openvpn-auth-oauth2 will use the token from the authorized user and fetch groups.

It cloud possible, that (external) users has only permissions to fetch group memberships they belong too.

The behavior should be reproducible with the API explorer. https://cloud.google.com/identity/docs/reference/rest/v1/groups.memberships/list . external users should use the API explorer, too.

[Pionerd]

You are right. When I do that call (as external), I get exactly the same permission denied as in the logs earlier. So it seems we may have hit a dead end here :(

[jkroepke]

You get permission denied for all groups?

[Pionerd]

Yes

[jkroepke]

Could you ask the Google support, if this is expectable and may ask how they would resolve the issue?

[Pionerd]

I'm sorry, unfortunately I do not have access to project with plans that allow creating cases...

[jkroepke]

I'm aware that just re-implement the service account approch would be the easiest solution how your. But from openvpn-auth-oauth2 point of view, a lot of dependencies are manatory to support all the special Google IAM Workflows, e.g. Default Credentials and SA Deligation. Additionally, such code is not easy to maintain, since it always require manual testing.

To continue investigate your issue, Google provides a Troubleshoot IAM permissions service. Maybe it make transparent where is thee is. But I have no idea, if it works with OAuth2 Apps in between.

https://cloud.google.com/policy-intelligence/docs/troubleshoot-access

[Pionerd]

Last week, we had an internal non-admin user who was also not able to retrieve his membership (same error), so we disabled group validation for now.

[jkroepke]

In Microsoft Entra ID, it possible to restrict the usage of an application. ref: https://learn.microsoft.com/en-us/entra/identity/enterprise-apps/what-is-access-management#assigning-users-and-groups-to-an-app

Instead having a group validation on application side, its possible to restrict who is allowed to use the application directly inside Entra ID.

No idea, if this concept exists in Google Workspace, too.

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[swetli]

I had the same issue: group validation worked for some members and not for others. I found the root cause - basically some members did not have enough permission to view the group membership hence got 403 - forbidden:

More precisely - openvpn-auth-oauth2[62293]: time=2024-08-28T10:13:30.721Z level=WARN msg="user validation: error from Google API https://cloudidentity.googleapis.com/v1/groups//memberships: http status code: 403; message: Error(4001): Permission denied for membership resource 'groups/' (or it may not exist)."

To solve the problem adjust the members permissions as shown below:

The settings are available under group settings in the admin.google.com console. I hope this helps

[jkroepke]

Great! How you please explain this more precisely?

[jkroepke]

I found it, it's the same setting as in the Admin Admin: https://admin.google.com/ac/groups//settings

However, it's impossible to permit "external" users here.

[swetli]

That is correct - even if one can add external users to a group, they are not allowed to view group members, hence this API call will always be rejected for them: