Google Login Not Working

[akath19]

Bug

Using Google Login always results in "forbidden" in the frontend and users are unable to login/signup with it.

Deployment Details

• Pyroscope Version: 0.23.0
• Deployment Type: Helm on GKE (chart version: 0.2.55)
• Network Type: GKE Ingress protected with Identity-Aware Proxy

Steps to Reproduce

1. Configure Google auth in config file:

   auth:
   google:
   enabled: true
   client-id: {REDACTED}
   client-secret: {REDACTED}
   allowed-domains:
   - example.com

2. Restart pyroscope with debug logs enabled
3. Click on "Sign in with Google" button in login screen
4. login screen redirects to "forbidden"

Diagnostic Details

Here are the full (redacted) logs for the auth attempt:

10.10.20.15 - - [27/Jul/2022:20:20:23 +0000] "GET /auth/google/login?tls=true HTTP/1.1" 307 405
10.10.20.15 - - [27/Jul/2022:20:20:24 +0000] "GET /auth/google/callback?state={redacted}&code={redacted}&scope=email+openid+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email&authuser=0&hd=example.com&prompt=none HTTP/1.1" 308 330
time="2022-07-27T20:20:24.295453" level=error msg="failed to generate oauth client" file=" server/login.go:213" error="exchanging auth code for token failed: oauth2: cannot fetch token: 400 Bad Request\nResponse: {\n  \"error\": \"invalid_grant\",\n  \"error_description\": \"Bad Request\"\n}"
10.10.20.15 - - [27/Jul/2022:20:20:24 +0000] "GET /auth/google/redirect?state={redacted}&code={redacted}&scope=email+openid+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email&authuser=0&hd=example.com&prompt=none&tls=false HTTP/1.1" 307 46
time="2022-07-27T20:20:24.402796" level=debug msg="unauthenticated request" file=" api/auth.go:51" method=GET remote="10.10.20.15:53751" url=/forbidden
10.10.20.15 - - [27/Jul/2022:20:20:24 +0000] "GET /forbidden HTTP/1.1" 200 1190
time="2022-07-27T20:20:27.117584" level=debug msg="unauthenticated request" file=" api/auth.go:51" method=GET remote="10.10.20.15:51341" url=/api/user
time="2022-07-27T20:20:27.117644" level=debug msg=unauthorized file=" server/httputils/default.go:112" code=401 error="invalid credentials" method=GET remote="10.10.20.15:51341" url=/api/user

The only difference I can see from other Google OAuth implementations is that the https://www.googleapis.com/auth/userinfo.email scope (set here) isn't sent by the client, also, that specific scope isn't available in the Google OAuth scopes page

[akath19]

I'll be happy to submit a PR for this is my reasoning is correct!

[petethepig]

Thanks for the detailed report! I can't reproduce. We use google oauth when we run pyroscope internally and we haven't seen this issue before.

Go ahead with the PR if you want to give it a try and we'd love to merge it if it fixes the issue for you!

Can you tell us a bit more about "GKE Ingress protected with Identity-Aware Proxy"? Maybe that is the reason for this behavior. Is this a reverse proxy to pyroscope server? If it is and it's already providing authentication, in that case you don't need to enable authentication on pyroscope side.

[akath19]

@petethepig I debugged this some more and it turns out I needed to add the redirect-uri config variable which isn't documented in the docs site. Once I added it, everything worked correctly.

Regarding Identity-Aware Proxy, it's basically an authentication proxy by Google that requires a user be in a certain Google group to be able to access the UI for an application deployed with a GKE ingress, I can't disable auth on pyroscope because my org requires authentication both for IAP and for the specific application behind it.

[amontalban]

@petethepig I debugged this some more and it turns out I needed to add the redirect-uri config variable which isn't documented in the docs site. Once I added it, everything worked correctly.

Regarding Identity-Aware Proxy, it's basically an authentication proxy by Google that requires a user be in a certain Google group to be able to access the UI for an application deployed with a GKE ingress, I can't disable auth on pyroscope because my org requires authentication both for IAP and for the specific application behind it.

@akath19 how did you get it working? Are you using latest version? Because I tried with this config file and it does not work for me:

log-level: "debug"
auth:
  signup-default-role: Admin
  login-maximum-lifetime-days: 7
  google:
    enabled: true
    client-id: "CLIENT_ID_VALUE"
    client-secret: "CLIENT_SECRET_VALUE"
    redirect-url: "https://pyroscope.domain.com/auth/google/callback"
    allowed-domains:
      - domain.com

Looks like redirect-url is deprecated and did not work for me, and this is what I see in the logs:

10.222.20.17 - - [03/Jun/2023:14:05:42 +0000] "GET /auth/google/callback?state=b38616383526ba65788cb5def2d5851c&code=4%2F0AbUR2VMxCj9OhKz-SIl4EVa2XF4G1D0TwTvM3_d7tekhhLvQOcJ86KimJGTxY5l0Sre-Nw&scope=email+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+openid&authuser=0&hd=domain.com&prompt=none HTTP/1.1" 308 323
time="2023-06-03T14:05:43.353873" level=error msg="missing state cookie" file=" server/login.go:213" error="http: named cookie not present"
10.222.21.10 - - [03/Jun/2023:14:05:43 +0000] "GET /auth/google/redirect?state=b38616383526ba65788cb5def2d5851c&code=4%2F0AbUR2VMxCj9OhKz-SIl4EVa2XF4G1D0TwTvM3_d7tekhhLvQOcJ86KimJGTxY5l0Sre-Nw&scope=email+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+openid&authuser=0&hd=domain.com&prompt=none&tls=false HTTP/1.1" 307 46
time="2023-06-03T14:05:43.619573" level=debug msg="flushing current batch" file=" storage/storage_exemplars.go:145"
10.222.21.10 - - [03/Jun/2023:14:05:43 +0000] "GET /forbidden HTTP/1.1" 404 1149
10.222.20.17 - - [03/Jun/2023:14:05:44 +0000] "GET /index.html HTTP/1.1" 404 1149

Any help is highly appreciated!