search

calcom / cal.com

Scheduling infrastructure for absolutely everyone.
https://cal.com
Other
32.34k stars 7.97k forks source link

Blocked Google Calendar requests due to `invalid_grant` from expired token #5796

Open eeshaan opened 1 year ago

eeshaan commented 1 year ago

Issue Summary

When stopping the Cal.com web instance and restarting it, the Google Calendar response comes back as with a 400 bad request. Instead of attempted to fetch a new token, the Google Calendar integration breaks until the user removes the calendar connection and signs back into their Google account.

Steps to Reproduce

  1. Connect Google Calendar OAuth Client
  2. Restart @calcom/web
  3. invalid_grant on Google Calendar
  4. User has to remove calendar connection and sign back into the Google account.

Technical details

cal.com v2.3.2 — though the issue has been present in previous releases as well.

@calcom/web:start: 04:43:04.176 timeZoneName WARN CalendarManager
@calcom/web:start:  Error  invalid_grant
@calcom/web:start: details:
@calcom/web:start: <ref *1> {
@calcom/web:start:   response: {
@calcom/web:start:     config: {
@calcom/web:start:       method: 'POST',
@calcom/web:start:       url: 'https://oauth2.googleapis.com/token',
@calcom/web:start:       data: 'refresh_token=[redacted]&client_id=[redacted].apps.googleusercontent.com&client_secret=[redacted]&grant_type=refresh_token',
@calcom/web:start:       headers: {
@calcom/web:start:         'Content-Type': 'application/x-www-form-urlencoded',
@calcom/web:start:         'User-Agent': 'google-api-nodejs-client/7.14.1',
@calcom/web:start:         'x-goog-api-client': 'gl-node/16.18.0 auth/7.14.1',
@calcom/web:start:         Accept: 'application/json'
@calcom/web:start:       },
@calcom/web:start:       paramsSerializer: [Function: paramsSerializer],
@calcom/web:start:       body: 'refresh_token=[redacted]&client_id=[redacted].apps.googleusercontent.com&client_secret=[redacted]&grant_type=refresh_token',
@calcom/web:start:       validateStatus: [Function: validateStatus],
@calcom/web:start:       responseType: 'json'
@calcom/web:start:     },
@calcom/web:start:     data: {
@calcom/web:start:       error: 'invalid_grant',
@calcom/web:start:       error_description: 'Token has been expired or revoked.'
@calcom/web:start:     },
@calcom/web:start:     headers: {
@calcom/web:start:       'cache-control': 'no-cache, no-store, max-age=0, must-revalidate',
@calcom/web:start:       connection: 'close',
@calcom/web:start:       'content-encoding': 'gzip',
@calcom/web:start:       'content-type': 'application/json; charset=utf-8',
@calcom/web:start:       date: 'Tue, 29 Nov 2022 04:43:04 GMT',
@calcom/web:start:       expires: 'Mon, 01 Jan 1990 00:00:00 GMT',
@calcom/web:start:       pragma: 'no-cache',
@calcom/web:start:       server: 'scaffolding on HTTPServer2',
@calcom/web:start:       'transfer-encoding': 'chunked',
@calcom/web:start:       vary: 'Origin, X-Origin, Referer',
@calcom/web:start:       'x-content-type-options': 'nosniff',
@calcom/web:start:       'x-frame-options': 'SAMEORIGIN',
@calcom/web:start:       'x-xss-protection': '0'
@calcom/web:start:     },
@calcom/web:start:     status: 400,
@calcom/web:start:     statusText: 'Bad Request',
@calcom/web:start:     request: {
@calcom/web:start:       responseURL: 'https://oauth2.googleapis.com/token'
@calcom/web:start:     }
@calcom/web:start:   },
@calcom/web:start:   config: [Circular *1],
@calcom/web:start:   code: '400'
@calcom/web:start: }
@calcom/web:start: error stack:
@calcom/web:start: • gaxios.ts:158 _request
@calcom/web:start:     node_modules/gaxios/src/gaxios.ts:158:15
@calcom/web:start:
@calcom/web:start: • task_queues:96 processTicksAndRejections
@calcom/web:start:     node:internal/process/task_queues:96:5
@calcom/web:start:
@calcom/web:start: • oauth2client.js:174 refreshTokenNoCache
@calcom/web:start:     node_modules/google-auth-library/build/src/auth/oauth2client.js:174:21
@calcom/web:start:
@calcom/web:start: • oauth2client.js:284 getRequestMetadataAsync
@calcom/web:start:     node_modules/google-auth-library/build/src/auth/oauth2client.js:284:17
@calcom/web:start:
@calcom/web:start: • oauth2client.js:357 requestAsync
@calcom/web:start:     node_modules/google-auth-library/build/src/auth/oauth2client.js:357:23
@calcom/web:start:
@calcom/web:start:
libeanim commented 1 year ago

We seem to have the same issue using calcom-docker (with cal.com v2.4.4). Were you able to fix it?

PeerRich commented 1 year ago

have you tried the latest here? https://hub.docker.com/r/calcom/cal.com/tags

libeanim commented 1 year ago

We've used the Dockerfile in the calcom/docker project to build our own image, as otherwise non-localhost domain names don't work as far as I understood.

PeerRich commented 1 year ago

right that makes sense. hmm that is odd

PeerRich commented 1 year ago

maybe @zomars knows more

zomars commented 1 year ago

No clue. It seems like the token gets invalidated. Maybe the encryption key resets on restart? Although that would mean password wouldn't work as well. @krumware any ideas?

krumware commented 1 year ago

I'm very curious as to the expiration and date portion of the request. Is this 'expires' value normal?

@calcom/web:start:       date: 'Tue, 29 Nov 2022 04:43:04 GMT',
@calcom/web:start:       expires: 'Mon, 01 Jan 1990 00:00:00 GMT',
krumware commented 1 year ago

Through some googling, this could be related to a number of configuration items, not necessarily those specific to the cal.com side. I've had problems in the past when the callback URL wasn't added properly, or my environmental variable had invalid characters or extra quotes.

Quick note, non-localhost domains do work via calcom/docker

libeanim commented 1 year ago

Thanks for looking into it! If its a configuration issue or an issue with the callback URL then it shouldn't work after reconnecting the google account, right? But that seems to work for now, as long as we don't restart or recreate the container.

@krumware re non-localhost domains, in the Readme it says:

...there are specific requirements for providing environmental variables at build-time in order to specify a non-localhost BASE_URL. ...

and also

For Production, for the time being, please checkout the repository and build/push your own image privately.

So it seems like we cannot use the pre-built image but need to build our own?

krumware commented 1 year ago

That's really interesting around the restart. We'll continue to monitor.

You can use it, it's not a license restriction or anything. It's just general best practice for container immutability to not allow some file writes at runtime, and some more compliance-heavy environments may have controls affecting that. So the true "enterprise production" recommendation would be to build it (until we fix that). But otherwise for most applications it's fine to run the provided container. We'll try to make that more clear!

libeanim commented 1 year ago

Ok it seems that the restarting (or recreating) the container has nothing to do with it! Its that the google tokens cannot get refreshed. Had it running w/o restart and after roughly one week the google tokens expire and I see this error:

calcom    | @calcom/web:start: 22:16:18.547 ERROR [[lib] google_calendar Error refreshing google token 
calcom    | @calcom/web:start:  Error  invalid_grant
calcom    | @calcom/web:start: details:
calcom    | @calcom/web:start: <ref *1> {
calcom    | @calcom/web:start:   response: {
calcom    | @calcom/web:start:     config: {
calcom    | @calcom/web:start:       method: 'POST',
calcom    | @calcom/web:start:       url: 'https://oauth2.googleapis.com/token',
calcom    | @calcom/web:start:       data: 'refresh_token=<redacted>&client_id=<redacted>.apps.googleusercontent.com&client_secret=<redacted>&grant_type=refresh_token',
calcom    | @calcom/web:start:       headers: {
calcom    | @calcom/web:start:         'Content-Type': 'application/x-www-form-urlencoded',
calcom    | @calcom/web:start:         'User-Agent': 'google-api-nodejs-client/7.14.1',
calcom    | @calcom/web:start:         'x-goog-api-client': 'gl-node/16.19.0 auth/7.14.1',
calcom    | @calcom/web:start:         Accept: 'application/json'
calcom    | @calcom/web:start:       },
calcom    | @calcom/web:start:       paramsSerializer: [Function: paramsSerializer],
calcom    | @calcom/web:start:       body: 'refresh_token=<redacted>&client_id=<redacted>.apps.googleusercontent.com&client_secret=<redacted>&grant_type=refresh_token',
calcom    | @calcom/web:start:       validateStatus: [Function: validateStatus],
calcom    | @calcom/web:start:       responseType: 'json'
calcom    | @calcom/web:start:     },
calcom    | @calcom/web:start:     data: {
calcom    | @calcom/web:start:       error: 'invalid_grant',
calcom    | @calcom/web:start:       error_description: 'Token has been expired or revoked.'
calcom    | @calcom/web:start:     },
calcom    | @calcom/web:start:     headers: {
calcom    | @calcom/web:start:       'alt-svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"',
calcom    | @calcom/web:start:       'cache-control': 'no-cache, no-store, max-age=0, must-revalidate',
calcom    | @calcom/web:start:       connection: 'close',
calcom    | @calcom/web:start:       'content-encoding': 'gzip',
calcom    | @calcom/web:start:       'content-type': 'application/json; charset=utf-8',
calcom    | @calcom/web:start:       date: 'Sun, 22 Jan 2023 22:16:18 GMT',
calcom    | @calcom/web:start:       expires: 'Mon, 01 Jan 1990 00:00:00 GMT',
calcom    | @calcom/web:start:       pragma: 'no-cache',
calcom    | @calcom/web:start:       server: 'scaffolding on HTTPServer2',
calcom    | @calcom/web:start:       'transfer-encoding': 'chunked',
calcom    | @calcom/web:start:       vary: 'Origin, X-Origin, Referer',
calcom    | @calcom/web:start:       'x-content-type-options': 'nosniff',
calcom    | @calcom/web:start:       'x-frame-options': 'SAMEORIGIN',
calcom    | @calcom/web:start:       'x-xss-protection': '0'
calcom    | @calcom/web:start:     },
calcom    | @calcom/web:start:     status: 400,
calcom    | @calcom/web:start:     statusText: 'Bad Request',
calcom    | @calcom/web:start:     request: {
calcom    | @calcom/web:start:       responseURL: 'https://oauth2.googleapis.com/token'
calcom    | @calcom/web:start:     }
calcom    | @calcom/web:start:   },
calcom    | @calcom/web:start:   config: [Circular *1],
calcom    | @calcom/web:start:   code: '400'
calcom    | @calcom/web:start: }
calcom    | @calcom/web:start: error stack:
calcom    | @calcom/web:start: • gaxios.ts:158 _request
calcom    | @calcom/web:start:     node_modules/gaxios/src/gaxios.ts:158:15
calcom    | @calcom/web:start: 
calcom    | @calcom/web:start: • task_queues:96 processTicksAndRejections
calcom    | @calcom/web:start:     node:internal/process/task_queues:96:5
calcom    | @calcom/web:start: 
calcom    | @calcom/web:start: • oauth2client.js:174 refreshTokenNoCache
calcom    | @calcom/web:start:     node_modules/google-auth-library/build/src/auth/oauth2client.js:174:21
calcom    | @calcom/web:start: 
calcom    | @calcom/web:start: • 7105.js:1116 refreshAccessToken
calcom    | @calcom/web:start:     .next/server/chunks/7105.js:1116:34
calcom    | @calcom/web:start: 
calcom    | @calcom/web:start: • 7105.js:1344 <anonymous>
calcom    | @calcom/web:start:     .next/server/chunks/7105.js:1344:3

The google credential environment variable looks like this:

GOOGLE_API_CREDENTIALS={"web":{"client_id":"<reacted>.apps.googleusercontent.com","project_id":"<redacted>","auth_uri":"https://accounts.google.com/o/oauth2/auth","token_uri":"https://oauth2.googleapis.com/token","auth_provider_x509_cert_url":"https://www.googleapis.com/oauth2/v1/certs","client_secret":"<redacted>","redirect_uris":["https://<redacted>/api/integrations/googlecalendar/callback","https://<redacted>/api/auth/callback/google"]}}

And the google console settings are: google_creds

The google cloud app is not published but in testing mode could that be a reason for this also? image

eeshaan commented 1 year ago

Ok it seems that the restarting (or recreating) the container has nothing to do with it! Its that the google tokens cannot get refreshed. Had it running w/o restart and after roughly one week the google tokens expire

+1 on this. Even though the issue is most easily replicated with a restart, I've also observed the same behavior after some arbitrary amount of runtime (also around 1 week). Unsure on this, but I think one possibility is that the token expires when server usage spikes and the app is briefly inaccessible.

eeshaan commented 1 year ago

The google cloud app is not published but in testing mode could that be a reason for this also?

The same is also true in our case.

krumware commented 1 year ago

That's good to know!

tarkilhk commented 2 months ago

Apologies for reviving the topic, but I am facing the same issue, and I can't find any other mention of how to deal with this, so I'm trying to see if anyone ever found a way to tackle this please ? Having to relink my calendars every few days is not really sustainable, so I imagine some people have found a root cause or a workaround ?

This is the error I see on my end fyi:

@calcom/web:start: 14:28:02:982 DEBUGapp-store/_utils/oauth/OAuthManager handleFetchError Error "Error: invalid_grant
    at Gaxios._request (/calcom/node_modules/gaxios/build/src/gaxios.js:129:23)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async MyGoogleAuth.refreshTokenNoCache (/calcom/node_modules/google-auth-library/build/src/auth/oauth2client.js:174:21)
    at async OAuthManager.fetchNewTokenObject (/calcom/apps/web/.next/server/chunks/72628.js:1:1244)
    at async OAuthManager.refreshOAuthToken (/calcom/apps/web/.next/server/chunks/15240.js:1:5848)
    at async OAuthManager.getTokenObjectOrFetch (/calcom/apps/web/.next/server/chunks/15240.js:1:2339)
    at async Object.getMyGoogleAuthWithRefreshedToken (/calcom/apps/web/.next/server/chunks/72628.js:1:2059)
    at async GoogleCalendarService.authedCalendar (/calcom/apps/web/.next/server/chunks/72628.js:1:2242)
    at async GoogleCalendarService.listCalendars (/calcom/apps/web/.next/server/chunks/72628.js:1:10030)
    at async /calcom/apps/web/.next/server/chunks/44121.js:1:1968"

@calcom/web:start: 14:28:02:985 DEBUGapp-store/_utils/oauth/OAuthManager refreshOAuthToken Response from refreshOAuthToken {"text":"{\"myFetchError\":\"invalid_grant\"}","ok":false,"status":500,"statusText":""}

@calcom/web:start: 14:28:02:990 ERRORapp-store/_utils/oauth/OAuthManager refreshOAuthToken Token parsing error: [{"code":"invalid_type","expected":"string","received":"undefined","path":["access_token"],"message":"Required"}]

Thanks vm in advance !

lesbass commented 2 months ago

I had the same issue and in the end I switchted to cal.com directly avoiding hosting it on my server :(

tarkilhk commented 1 month ago

I tried to look into the code, google, etc. It seems that it's because Google Calendar -> CalendarService.ts somehow doesn't detect properly that a token is expired, and tries to refresh it after its expiry date here:

    fetchNewTokenObject: async ({ refreshToken }: { refreshToken: string | null }) => { 
    const myGoogleAuth = await this.getMyGoogleAuthSingleton();
    const fetchTokens = await myGoogleAuth.refreshToken(refreshToken);   
    // Create Response from fetchToken.res   
    const response = new Response(JSON.stringify(fetchTokens.res?.data ?? null), {   
      status: fetchTokens.res?.status,   
      statusText: fetchTokens.res?.statusText,   
    });   
    return response;

I'm (really) no expert, but should there maybe be a check before calling myGoogleAuth.refreshToken() that token is actually not expired ? I'm getting out of ideas, not sure how this is not more widespread, I imagine a lot of people self host, and a lot of people use google calendar, so it's really surprising to me that not more people seem to face this ?

tarkilhk commented 1 month ago

Also it happens after 1 week exactly, because the tokens granted by Google oAuth are valid for 1 week (I checked epoch from the logs), which is why this starts happening after a week of usage.

I am also having App Status = Testing, as I haven't published it. Might it be that "Testing" tokens have 1 week duration while "Published" app ones don't have ? That would be a bit surprising, but who knows. In any case, I think handling expired tokens should work. Need to ask for a new one instead of refreshing existing one.

lesderid commented 1 month ago

From Using OAuth 2.0 to Access Google APIs (official Google docs):

A Google Cloud Platform project with an OAuth consent screen configured for an external user type and a publishing status of "Testing" is issued a refresh token expiring in 7 days, unless the only OAuth scopes requested are a subset of name, email address, and user profile (through the userinfo.email, userinfo.profile, openid scopes, or their OpenID Connect equivalents).

This essentially means that if your Google OAuth consent screen has the "Testing" status, the user has to re-authenticate weekly for it to keep working.

If you're fine with scary warnings when people link their Google Calendar, a possible solution is to click 'Publish app' so it enters "Production" (but unverified) status.

tarkilhk commented 1 month ago

Thank you so much for taking the time to check. I tried that before, but was put off by the verification process where Google rejected my app (as it's for a single user...). But thanks to you I just realised that I can publish it (and set it to Production), while still staying unverified.

I have now high hopes this should work, so thanks again very much for taking the time to comment. I still have a feeling that it should be feasible to refresh a token before it expires to keep it alive, but if this works, well, this works ! ^^

lesderid commented 1 month ago

I have now high hopes this should work, so thanks again very much for taking the time to comment.

Me too, it's just not clear to me yet that this will keep working indefinitely.

You're welcome!

I still have a feeling that it should be feasible to refresh a token before it expires to keep it alive, but if this works, well, this works ! ^^

No: there are two types of tokens: auth tokens and refresh tokens. Auth tokens are used to authenticate a user with the actual API and are always shortlived, but can be refreshed with a refresh token. This refresh token is supposed to work indefinitely (unless the user removes the connection), unless the application is in 'Testing' mode, in which case they are only valid for one week (and can only be removed and recreated by having the user log in again with their Google account).

pmffromspace commented 4 weeks ago

I think this workaround should be added to the docs 😃

tarkilhk commented 2 weeks ago

Well, 2 weeks later, I wanted to confirm that this solved the issue.

@lesderid , just wanted to say thank you again for taking the time to answer, and give detailed explanations about types of tokens, etc.

Really appreciated ! 👍