Netbird self-hosted with Google Workspace IdP: Error 400: invalid_scope

[redecs]

Describe the problem

Hello! I’m trying to setup Netbird self-hosted with Google Workspace IdP following the instructions outlined here: https://docs.netbird.io/selfhosted/identity-providers#google-workspace I’ve gone through the process multiple times only to encounter the same error when trying to access the Netbird Dashboard: I get redirected to Google Auth page with the error message Error 400: invalid_scope

Some requested scopes were invalid. {invalid=[netbird@<PROJECT_ID>.iam.gserviceaccount.com]} 

Tried searching around for similar issues and solutions but didn’t manage to find anything helpful.

To Reproduce

Steps to reproduce the behavior: Followed the steps from the guide: https://docs.netbird.io/selfhosted/identity-providers#google-workspace

Here's how my setup.env looks like:

## example file, you can copy this file to setup.env and update its values
##

# Image tags
# you can force specific tags for each component; will be set to latest if empty
NETBIRD_DASHBOARD_TAG=""
NETBIRD_SIGNAL_TAG=""
NETBIRD_MANAGEMENT_TAG=""
COTURN_TAG=""

# Dashboard domain. e.g. app.mydomain.com
NETBIRD_DOMAIN="app.mydomain.com"

# TURN server domain. e.g. turn.mydomain.com
# if not specified it will assume NETBIRD_DOMAIN
NETBIRD_TURN_DOMAIN=""

# TURN server public IP address
# required for a connection involving peers in
# the same network as the server and external peers
# usually matches the IP for the domain set in NETBIRD_TURN_DOMAIN
NETBIRD_TURN_EXTERNAL_IP="1.2.3.4"

# -------------------------------------------
# OIDC
#  e.g., https://example.eu.auth0.com/.well-known/openid-configuration
# -------------------------------------------
NETBIRD_AUTH_OIDC_CONFIGURATION_ENDPOINT="https://accounts.google.com/.well-known/openid-configuration"
# The default setting is to transmit the audience to the IDP during authorization. However,
# if your IDP does not have this capability, you can turn this off by setting it to false.
#NETBIRD_DASH_AUTH_USE_AUDIENCE=false
NETBIRD_AUTH_AUDIENCE="OAUTH_CLIENT_ID.apps.googleusercontent.com"
# e.g. netbird-client
NETBIRD_AUTH_CLIENT_ID="OAUTH_CLIENT_ID.apps.googleusercontent.com"
# indicates the scopes that will be requested to the IDP
NETBIRD_AUTH_SUPPORTED_SCOPES="netbird@<PROJECT_ID>.iam.gserviceaccount.com"
# NETBIRD_AUTH_CLIENT_SECRET is required only by Google workspace.
NETBIRD_AUTH_CLIENT_SECRET="OAUTH_CLIENT_SECRET"
# if you want to use a custom claim for the user ID instead of 'sub', set it here
# NETBIRD_AUTH_USER_ID_CLAIM=""
# indicates whether to use Auth0 or not: true or false
NETBIRD_USE_AUTH0="false"
# if your IDP provider doesn't support fragmented URIs, configure custom
# redirect and silent redirect URIs, these will be concatenated into your NETBIRD_DOMAIN domain.
# NETBIRD_AUTH_REDIRECT_URI="/peers"
# NETBIRD_AUTH_SILENT_REDIRECT_URI="/add-peers"
NETBIRD_AUTH_REDIRECT_URI="/auth"
NETBIRD_AUTH_SILENT_REDIRECT_URI="/silent-auth"
# Updates the preference to use id tokens instead of access token on dashboard
# Okta and Gitlab IDPs can benefit from this
NETBIRD_TOKEN_SOURCE="idToken"
# -------------------------------------------
# OIDC Device Authorization Flow
# -------------------------------------------
NETBIRD_AUTH_DEVICE_AUTH_PROVIDER="none"
NETBIRD_AUTH_DEVICE_AUTH_CLIENT_ID=""
# Some IDPs requires different audience, scopes and to use id token for device authorization flow
# you can customize here:
NETBIRD_AUTH_DEVICE_AUTH_AUDIENCE=$NETBIRD_AUTH_AUDIENCE
NETBIRD_AUTH_DEVICE_AUTH_SCOPE="openid"
NETBIRD_AUTH_DEVICE_AUTH_USE_ID_TOKEN=false
# -------------------------------------------
# OIDC PKCE Authorization Flow
# -------------------------------------------
# Comma separated port numbers. if already in use, PKCE flow will choose an available port from the list as an alternative
# eg. 53000,54000
NETBIRD_AUTH_PKCE_REDIRECT_URL_PORTS="53000"
# -------------------------------------------
# IDP Management
# -------------------------------------------
# eg. zitadel, auth0, azure, keycloak
NETBIRD_MGMT_IDP="google"
# Some IDPs requires different client id and client secret for management api
NETBIRD_IDP_MGMT_CLIENT_ID=$NETBIRD_AUTH_CLIENT_ID
NETBIRD_IDP_MGMT_CLIENT_SECRET=""
# Required when setting up with Keycloak "https://<YOUR_KEYCLOAK_HOST_AND_PORT>/admin/realms/netbird"
# NETBIRD_IDP_MGMT_EXTRA_ADMIN_ENDPOINT=
# With some IDPs may be needed enabling automatic refresh of signing keys on expire
# NETBIRD_MGMT_IDP_SIGNKEY_REFRESH=false
# NETBIRD_IDP_MGMT_EXTRA_ variables. See https://docs.netbird.io/selfhosted/identity-providers for more information about your IDP of choice.
NETBIRD_IDP_MGMT_EXTRA_SERVICE_ACCOUNT_KEY="BASE64_ENCODED_KEY"
NETBIRD_IDP_MGMT_EXTRA_CUSTOMER_ID="GOOGLE_WORKSPACE_CUSTOMER_ID"
# -------------------------------------------
# Letsencrypt
# -------------------------------------------
# Disable letsencrypt
#  if disabled, cannot use HTTPS anymore and requires setting up a reverse-proxy to do it instead
NETBIRD_DISABLE_LETSENCRYPT=false
# e.g. hello@mydomain.com
NETBIRD_LETSENCRYPT_EMAIL="hello@mydomain.com"

Expected behavior

I would have expected the Google authorization to succeed.

Are you using NetBird Cloud?

No, I'm using the self-host NetBird's control plane.

NetBird version

v0.27.7 (git tag of the repo with the tools I've used for the setup), latest in the generated docker-compose.yml

Additional context

Before trying the Advanced Setup with Google Workspace as IdP I tested Netbird with the Quick Start Guid with Zitadel and that worked fine but I would like to remove to remove additional component (Zitadel) and use Google as the IdP.

Thank you!

[redecs]

Some additional information: the management services keeps restarting, here is the log snippet that keeps repeating:

management-1  | 2024-05-20T20:03:03Z INFO management/cmd/management.go:455: loading OIDC configuration from the provided IDP configuration endpoint https://accounts.google.com/.well-known/openid-configuration
management-1  | 2024-05-20T20:03:04Z INFO management/cmd/management.go:460: loaded OIDC configuration from the provided IDP configuration endpoint: https://accounts.google.com/.well-known/openid-configuration
management-1  | 2024-05-20T20:03:04Z INFO management/cmd/management.go:462: overriding HttpConfig.AuthIssuer with a new value https://accounts.google.com, previously configured value: https://accounts.google.com
management-1  | 2024-05-20T20:03:04Z INFO management/cmd/management.go:466: overriding HttpConfig.AuthKeysLocation (JWT certs) with a new value https://www.googleapis.com/oauth2/v3/certs, previously configured value: https://www.googleapis.com/oauth2/v3/certs
management-1  | 2024-05-20T20:03:04Z INFO management/server/telemetry/app_metrics.go:177: enabled application metrics and exposing on http://0.0.0.0:8081
management-1  | 2024-05-20T20:03:04Z INFO management/server/store.go:95: using SQLite store engine
management-1  | 2024-05-20T20:03:04Z INFO management/server/migration/migration.go:128: No records in table peers, no migration needed
management-1  | 2024-05-20T20:03:04Z INFO management/server/migration/migration.go:128: No records in table peers, no migration needed
management-1  | 2024-05-20T20:03:04Z INFO management/cmd/management.go:173: geo location service has been initialized from /var/lib/netbird/
management-1  | 2024-05-20T20:03:04Z INFO management/server/account.go:889: single account mode enabled, accounts number 0
management-1  | Error: failed creating JWT validator: invalid character '<' looking for beginning of value

Tracked down the error here: https://github.com/netbirdio/netbird/blob/main/management/cmd/management.go#L242

[redecs]

After more digging I discovered that the GET request for https://www.googleapis.com/oauth2/v3/certs fails when performed insider the container. Tested with go code (similar to Netbird) and curl. From inside the container I get a 403, while if I do it on the host I get expected 200 response.

[redecs]

At this point I'm pretty sure this issue is related to some networking issues, either related to Docker or Hetzner (the provider for the VPS) or a combination of those two, so I'm closing this.

[redecs]

I've setup a different VPS where the networking isn't an issue anymore and now all the docker compose services are running fine (no restarts) but unfortunately the Google Workspace Issue is still there.

This is how the manager logs look now:

management-1  | 2024-05-21T11:24:45Z INFO management/cmd/management.go:292: running gRPC backward compatibility server: [::]:33073
management-1  | 2024-05-21T11:24:45Z INFO management/cmd/management.go:324: management server version 0.27.7
management-1  | 2024-05-21T11:24:45Z INFO management/cmd/management.go:325: running HTTP server and gRPC server on the same port: [::]:443
management-1  | 2024-05-21T11:24:45Z INFO management/server/account.go:1076: 1 entries received from IdP management
management-1  | 2024-05-21T11:24:45Z INFO management/server/account.go:1107: warmed up IDP cache with 0 entries for 0 accounts

[pascal-fischer]

Hi @redecs, NETBIRD_AUTH_SUPPORTED_SCOPES="netbird@<PROJECT_ID>.iam.gserviceaccount.com" should be NETBIRD_AUTH_SUPPORTED_SCOPES="openid profile email"

[redecs]

Hi @pascal-fischer, that worked! Now that you point it to me, it totally makes sense. Thank you!