IDP Token Issue when using Google

drew-viles commented 5 months ago

Hi!

I'm using the latest Helm chart for the operator and tenant and I'm switching from using Authentik as my OIDC IDP over to Google.

Using Authentik, everything work perfectly but upon switching to Google, I'm getting the following error when logging in:

Error from IDP
An error occurred, please try again
Token used before issued

I know I'm probably going to get some response here about fixing NTP etc but this is on a Kubernetes cluster where other services interact with Google just fine for OAuth and the only variable here is MinIO.

As I say, Google OAuth works fine on other services on the same cluster and logging into MinIO works fine when using Authentik as the IDP.

Expected Behavior

It should log in without a token issue.

Current Behavior

It doesn't log in. It's never successful. There are no useful errors in the logs - in fact nothing happens in the tenant log or the console log for the operator. I'd have hoped for something to be output to help with diganosis of such an issue.

Possible Solution

I don't have one unfortunately. Some more verbose logging would be nice though.

Steps to Reproduce (for bugs)

Setup Google as an IDP and test it - let me know if you see it working and if so, can your document this a little better in the documentation. I've got a set of client credentials via GCP, I'm using a Google Workspace user and as I say, this works for Vault, Grafana, ArgoCD and more on this cluster. I can only presume MinIO is doning something different?

Context

I can't log in with Google as an IDP but can with Authentik.

Regression

Not to my knowledge

Your Environment

Version used (minio-operator): 5.0.15
Environment name and version (e.g. kubernetes v1.17.2): K8S v1.29.2
Operating System and version (uname -a): Ubunutu 22.04
Link to your deployment file:

The important part of my helm chart (the ClientID and secret are passed in via a secret in the same way I did with Authentik.

...
            env:
              - name: MINIO_IDENTITY_OPENID_ENABLE
                value: "on"
              - name: MINIO_IDENTITY_OPENID_DISPLAY_NAME
                value: "Log in with Google"
              - name: MINIO_IDENTITY_OPENID_CONFIG_URL
                value: https://accounts.google.com/.well-known/openid-configuration
              - name: MINIO_IDENTITY_OPENID_CLAIM_NAME
                value: "groups"
              - name: MINIO_IDENTITY_OPENID_SCOPES
                value: "openid,profile,email,https://www.googleapis.com/auth/admin.directory.group.readonly"
              - name: MINIO_BROWSER_REDIRECT_URL
                value: "https://{{ include "apps.minio.consoleUrl" . }}"
...

Any assistance would be greatly appreciated as I'm at a bit of a loss now.

This doesn't even cover how I map a Google Workspace group to something like consoleAdmin (which I suspect is custom policies which cannot be defined via the chart (supported in the community one). But one step at a time :-)

Thanks for taking the time with this!

drew-viles commented 5 months ago

oh, this is happening with Authentik since switching back just now too - other systems on the cluster still work fine with OAuth against Authentik and Google

Let me dig into this my side. The containers have the correct date in terms of UTC - I'll check the nodes themselves and see whats going on there. If I find a solution I'll post it here and close this.

drew-viles commented 5 months ago

~Yup, 100% my side - sorry for the bug BUT if anyone comes across this now, here is what to check.~

Notice with timedatectl there is a mismatch between RTC and Local/Universal time. This was on one of my control plane nodes. All workers were fine and the time in the containers appeared correct. The problem was the RTC on this one node.

$ timedatectl
               Local time: Thu 2024-05-23 15:21:17 UTC
           Universal time: Thu 2024-05-23 15:21:17 UTC
                 RTC time: Thu 2024-05-23 15:20:51
                Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: no
              NTP service: active
          RTC in local TZ: no

To fix it I did:

$ hwclock -w

Then the clocks were sync'd and tokens were issued correctly with valid times.

$ timedatectl
               Local time: Thu 2024-05-23 15:22:15 UTC
           Universal time: Thu 2024-05-23 15:22:15 UTC
                 RTC time: Thu 2024-05-23 15:22:16
                Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: no
              NTP service: active
          RTC in local TZ: no

drew-viles commented 5 months ago

Actually no, it's still doing it with Google but the above fixed my Authentik issues.

minio / operator