Self-hosted: google calendar refresh issue?

[piec]

Issue Summary

Hello,

When using calcom in a self-hosted docker-compose setup my availability based on a google calendar does not change when an event is added. It works on cal.com/ .

Steps to Reproduce

1. Setup a self-hosted docker-compose calcom with correct parameters (url, email, google credentials, ...)
2. Setup an account with a single Event Type of 60min (time slot intervals = 60min too).
3. Configure this account with google calendar, and toggle to enable a single calendar "X" in the "Check for conflicts" section. Use another calendar to add events to.
4. Check the availability using the event "preview" url, it correctly takes my google calendar events into account. 16:00 is available.
5. Add a new 60min event in google calendar at 16:00
6. Wait some time check the availability again, (refresh page even though it doesn't seem necessary). 16:00 is still erroneously available.
7. In the google calendar settings, toggle the calendar "X" 2 times to disable it and enable it again. (This step seems to force the refresh of the google calendar).
8. Check the availability, 16:00 is not available anymore which is correct.

The exact same setup with the hosted version at https://cal.com works. I suspect a calendar cache invalidation issue but I don't know the internals of calcom. What do you think? Thanks

Debug info during step 6

cal-calcom-1  | @calcom/web:start: https://cal.example.com/_next/data/.../en/pierre/calendar-cache/2023-03.json?user=pierre&month=2023-03
cal-calcom-1  | @calcom/web:start: https://cal.example.com/_next/data/.../en/pierre/calendar-cache/2023-03.json?user=pierre&month=2023-03
cal-calcom-1  | @calcom/web:start: Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
cal-calcom-1  | @calcom/web:start:     at new NodeError (node:internal/errors:387:5)
cal-calcom-1  | @calcom/web:start:     at ServerResponse.setHeader (node:_http_outgoing:644:11)
cal-calcom-1  | @calcom/web:start:     at ServerResponse._res.setHeader (/calcom/node_modules/next/dist/server/base-server.js:129:24)
cal-calcom-1  | @calcom/web:start:     at Function.getInitialProps (/calcom/apps/web/.next/server/pages/_document.js:127:22)
cal-calcom-1  | @calcom/web:start:     at Object.<anonymous> (/calcom/node_modules/next/dist/shared/lib/utils.js:84:33)
cal-calcom-1  | @calcom/web:start:     at Generator.next (<anonymous>)
cal-calcom-1  | @calcom/web:start:     at asyncGeneratorStep (/calcom/node_modules/@swc/helpers/lib/_async_to_generator.js:13:28)
cal-calcom-1  | @calcom/web:start:     at _next (/calcom/node_modules/@swc/helpers/lib/_async_to_generator.js:31:17)
cal-calcom-1  | @calcom/web:start:     at /calcom/node_modules/@swc/helpers/lib/_async_to_generator.js:36:13
cal-calcom-1  | @calcom/web:start:     at new Promise (<anonymous>)
cal-calcom-1  | @calcom/web:start:     at Object.<anonymous> (/calcom/node_modules/@swc/helpers/lib/_async_to_generator.js:28:16)
cal-calcom-1  | @calcom/web:start:     at Object.loadGetInitialProps (/calcom/node_modules/next/dist/shared/lib/utils.js:62:33)
cal-calcom-1  | @calcom/web:start:     at loadDocumentInitialProps (/calcom/node_modules/next/dist/server/render.js:692:48)
cal-calcom-1  | @calcom/web:start:     at renderDocument (/calcom/node_modules/next/dist/server/render.js:749:45)
cal-calcom-1  | @calcom/web:start:     at Object.renderToHTML (/calcom/node_modules/next/dist/server/render.js:786:34)
cal-calcom-1  | @calcom/web:start:     at processTicksAndRejections (node:internal/process/task_queues:96:5) {
cal-calcom-1  | @calcom/web:start:   code: 'ERR_HTTP_HEADERS_SENT'
cal-calcom-1  | @calcom/web:start: }

the https://cal.example.com/_next/data/.../en/pierre/calendar-cache/2023-03.json?user=pierre&month=2023-03 link shows the not-updated data where my slot at 16:00 is still available. The date field timestamp is before the time where the event was added in step 5.

Technical details

• browsers: chrome, firefox
• calcom versions: v2.6.6, main

[laxmikanta415]

@PeerRich can i look into this issue ?

[PeerRich]

go for it @laxmikanta415

[laxmikanta415]

Hi @piec I have tried to reproduce the issue following your steps, but not able to reproduce it. So to see if i have missed any steps let me know, i have recorded the scenario here:

https://www.loom.com/share/e085c3759bcc4adc9147ee3b556299ce

[sypion]

I too am also having this issue. Simply just being on a booking page that I'm self-hosting off a clean slate reproduces this issue. The error appears every few seconds.

@calcom/web:start: Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
@calcom/web:start:     at new NodeError (node:internal/errors:372:5)
@calcom/web:start:     at ServerResponse.setHeader (node:_http_outgoing:576:11)
@calcom/web:start:     at ServerResponse._res.setHeader (/Users/user/cal/node_modules/next/dist/server/base-server.js:180:24)
@calcom/web:start:     at Function.getInitialProps (/Users/user/cal/apps/web/.next/server/pages/_document.js:126:22)
@calcom/web:start:     at Object.<anonymous> (/Users/user/cal/node_modules/next/dist/shared/lib/utils.js:84:33)
@calcom/web:start:     at Generator.next (<anonymous>)
@calcom/web:start:     at asyncGeneratorStep (/Users/user/cal/node_modules/@swc/helpers/lib/_async_to_generator.js:13:28)
@calcom/web:start:     at _next (/Users/user/cal/node_modules/@swc/helpers/lib/_async_to_generator.js:31:17)
@calcom/web:start:     at /Users/user/cal/node_modules/@swc/helpers/lib/_async_to_generator.js:36:13
@calcom/web:start:     at new Promise (<anonymous>)
@calcom/web:start:     at Object.<anonymous> (/Users/user/cal/node_modules/@swc/helpers/lib/_async_to_generator.js:28:16)
@calcom/web:start:     at Object.loadGetInitialProps (/Users/user/cal/node_modules/next/dist/shared/lib/utils.js:62:33)
@calcom/web:start:     at loadDocumentInitialProps (/Users/user/cal/node_modules/next/dist/server/render.js:682:48)
@calcom/web:start:     at renderDocument (/Users/user/cal/node_modules/next/dist/server/render.js:732:45)
@calcom/web:start:     at /Users/user/cal/node_modules/next/dist/server/render.js:769:114
@calcom/web:start:     at NextTracerImpl.trace (/Users/user/cal/node_modules/next/dist/server/lib/trace/tracer.js:56:20)

I am on Node version 16.6.0

[laxmikanta415]

@sypion are you facing the issue with your busy times of google calendar not being excluded from availability OR its just this error in the log ?

I cannot able to reproduce this error as you can see from the below screenshot, let me know if there is a step that you are taking that i am missing.

[sypion]

This is an error that appears after reaching (for example) cal.example.com/sypion/event. Editing the event doesn't actually refresh the booking page either. Tempted to just clear the calendso database on my Postgres server and see if that helps but wanted to explore other solutions if possible.

[piec]

Hi @piec I have tried to reproduce the issue following your steps, but not able to reproduce it. So to see if i have missed any steps let me know, i have recorded the scenario here:

https://www.loom.com/share/e085c3759bcc4adc9147ee3b556299ce

Hi @laxmikanta415 thanks for trying to reproduce this, weird. I watched the video and this is the same steps as me. I doubt the ERR_HTTP_HEADERS_SENT error is related, but did you get that error too in the logs?

This is an error that appears after reaching (for example) cal.example.com/sypion/event. Editing the event doesn't actually refresh the booking page either. Tempted to just clear the calendso database on my Postgres server and see if that helps but wanted to explore other solutions if possible.

I had the same reflection the other day before creating this issue and tried from scratch by changing the pg volume. But it did not help.

[laxmikanta415]

No @piec i am not getting that error in the log as well. That is why i am not sure where in the code i should debug for this issue.

My guess is that: in some API the response is first sent with out waiting for an async function. (Missing await) So after sending the response the API is trying to send more data after promise is successful . I just need a way to reproduce it so i can check my guess.

[piec]

Ok interesting, one thing I noticed I must be using a more powerful machine because I have much less latency/loading compared to your recorded video. I might influence the scheduling of tasks.

[funsurfer2008]

Hy. i have the same issue with my installation and an exchange onpremise. brgds

[laxmikanta415]

@piec interesting, there are some other applications running on my machine as well so there is this latency on my system.

[funsurfer2008]

@laxmikanta415 i have the same issue on my Server since i have updated from 1.4.x. for example if i make an entry in my exchange calendar nothing on the cache is changed in cal.com. if i toggle the calendar off and on it is doing a revalidation:

but if i go to the public booking page on the day it is not refreshing and give this error:

brgds

[laxmikanta415]

@funsurfer2008 thanks for sharing the screenshots. I am going to check if i can capture the error by going to the public page.

[devsyed]

Hey! I am facing the same issue, cal.com does sync all the events from google calendar when I integrate it but after its integrated, it never pulls the events added to google calendar, which is allowing duplicate bookings with cal.com.

Thanks.

[piec]

Hey! I am facing the same issue, cal.com does sync all the events from google calendar when I integrate it but after its integrated, it never pulls the events added to google calendar, which is allowing duplicate bookings with cal.com.

Hi @devsyed, on a self-hosted instance or on https://cal.com/ ?

[devsyed]

Hey! I am facing the same issue, cal.com does sync all the events from google calendar when I integrate it but after its integrated, it never pulls the events added to google calendar, which is allowing duplicate bookings with cal.com.

Hi @devsyed, on a self-hosted instance or on https://cal.com/ ?

Its a self-hosted instance.

[laxmikanta415]

@PeerRich is there a document about the logic of sync process of events from google calendar ?

[laxmikanta415]

@funsurfer2008 As per your comment i have tried toggling the calendar conflict check checkbox and observed the logs. But unfortunately i am not able to reproduce it on my side:

I tried to then view my availability on my event page: http://localhost:3000/userrname/60min?date=2023-03-09 it is considering my calendar events as busy hours and hiding those from availability as per expected behaviour.

@piec can you reproduce the issue on your side ?

[funsurfer2008]

@laxmikanta415 if you toggle the calendar it sync with my calendar. and then you can go to the public site. the avaailability is correct. but if you make a new entry in the calendar, the availability is not again synced with the calendar. to get it in sync you must toggle the calendar again.

so make a new entry in your exchange calendar and check if the availability is correct without toggle the calendar.

maybe the error we see in the logs is from the URL. you access from the localhost. i access the cal.com selfhosted from the external url.

brgds

[funsurfer2008]

@funsurfer2008 As per your comment i have tried toggling the calendar conflict check checkbox and observed the logs. But unfortunately i am not able to reproduce it on my side:

I tried to then view my availability on my event page: http://localhost:3000/userrname/60min?date=2023-03-09 it is considering my calendar events as busy hours and hiding those from availability as per expected behaviour.

@piec can you reproduce the issue on your side ?

@laxmikanta415 as i see, in the logs you have posted, that the entrys are from the calendar toggle. the second screen from the error is if i access the user page like you do from external (not localhost) and i think if the server is syncing the calendar.

brgds

[laxmikanta415]

@funsurfer2008 Please confirm if the following steps are correct to reproduce the error:

1. Run the docker instance with calendar key set
2. Connect google calendar and enable check for conflict
3. Go to view your availability page for a specific date: http://localhost:3000/userrname/60min?date=2023-03-09
4. Now add some time to your google calendar ex: 10am to 11am
5. Now go to page of step 3 (i.e. availability page)

As per the error i should observe two things:

1. Error log in console with the error ERR_HTTP_HEADER_SENT
2. My availability should include 10am to 11am to my availability which is not correct.

[funsurfer2008]

@laxmikanta415 i have installed it without docker. 1) run Yarn Start 2) connect Exchange 2016 oder Exchange with my Account 3) Go to my availability page for a specific date (book.a-square.at not localhost) 4) add time to my outlook calendar 5) check in my availability page for a specific date (result:not here) 6) go to my settings and toggle calendar off/on 7) check in my availability page for a specific date (result: can see that the entry is synced an no timeslot is available) 8) go to my outlook and move the entry 9) check in my availability page for a specific date (result: availability is not changed) 10) repeat step 6 - 9 to reproduce.

brgds

[laxmikanta415]

@funsurfer2008 this ticket is for google calendar issue and on the self-hosted version, so i am not sure if the two issues are connected.

@PeerRich should there be a separate issue created for this, so that the problem is scoped.

[funsurfer2008]

@laxmikanta415 i think its not google or MS related. i have the same error as described in first post. mybe its a sync problem from self hosted CAL

[piec]

I agree that we don't necessarily need another issue, it could be an issue at higher level than specific calendar client implementations. By the way, I just took a look and I'm seeing CalendarService.ts, CalendarManager.ts files. Is there a function that is expected to run that does the calendar refresh where I could add a debug log? Are there debug flags that I can enable?

[laxmikanta415]

@piec @funsurfer2008 can someone send a loom screencast of the issue as i think these may be specific to certain calendar provider and also in specific deployment of the app. May be in localhost the same scenario is getting reproduced.

[PeerRich]

team is looking into it

[piec]

If it's still needed I can record a video

[sypion]

This issue is still persisting even after recent updates. Any news on this? 🙏

[laxmikanta415]

@PeerRich is someone from the team working on it still , if not send me a screencast of the error (may be loom) so i can debug and work on fixing it.

[piec]

Hello @sypion, @laxmikanta415, I've just pulled the latest calcom.docker.scarf.sh/calcom/cal.com:latest sha256:f9061508e4efdd2b00e20f1d0b8ab3b9435228627de953d6e9355ca5d6c8f10a.

I unfortunately still experience the same issue.

I'm seeing some @calcom/web:start: revalidating /pierre/calendar-cache/2023-03 style logs when I toggle a calendar but not later on.

Not sure that it's related but I'm also seeing logs like:

cal-calcom-1  | @calcom/web:start: Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
cal-calcom-1  | @calcom/web:start:     at new NodeError (node:internal/errors:387:5)
cal-calcom-1  | @calcom/web:start:     at ServerResponse.setHeader (node:_http_outgoing:644:11)
cal-calcom-1  | @calcom/web:start:     at ServerResponse._res.setHeader (/calcom/node_modules/next/dist/server/base-server.js:180:24)
cal-calcom-1  | @calcom/web:start:     at Function.getInitialProps (/calcom/apps/web/.next/server/pages/_document.js:126:22)
cal-calcom-1  | @calcom/web:start:     at Object.<anonymous> (/calcom/node_modules/next/dist/shared/lib/utils.js:84:33)
cal-calcom-1  | @calcom/web:start:     at Generator.next (<anonymous>)
cal-calcom-1  | @calcom/web:start:     at asyncGeneratorStep (/calcom/node_modules/@swc/helpers/lib/_async_to_generator.js:13:28)
cal-calcom-1  | @calcom/web:start:     at _next (/calcom/node_modules/@swc/helpers/lib/_async_to_generator.js:31:17)
cal-calcom-1  | @calcom/web:start:     at /calcom/node_modules/@swc/helpers/lib/_async_to_generator.js:36:13
cal-calcom-1  | @calcom/web:start:     at new Promise (<anonymous>)
cal-calcom-1  | @calcom/web:start:     at Object.<anonymous> (/calcom/node_modules/@swc/helpers/lib/_async_to_generator.js:28:16)
cal-calcom-1  | @calcom/web:start:     at Object.loadGetInitialProps (/calcom/node_modules/next/dist/shared/lib/utils.js:62:33)
cal-calcom-1  | @calcom/web:start:     at loadDocumentInitialProps (/calcom/node_modules/next/dist/server/render.js:682:48)
cal-calcom-1  | @calcom/web:start:     at renderDocument (/calcom/node_modules/next/dist/server/render.js:732:45)
cal-calcom-1  | @calcom/web:start:     at /calcom/node_modules/next/dist/server/render.js:769:114
cal-calcom-1  | @calcom/web:start:     at NextTracerImpl.trace (/calcom/node_modules/next/dist/server/lib/trace/tracer.js:56:20) {
cal-calcom-1  | @calcom/web:start:   code: 'ERR_HTTP_HEADERS_SENT'
cal-calcom-1  | @calcom/web:start: }

I will post a video later

[laxmikanta415]

@piec when you have time please post the video regarding this. I am not seeing any doc related to this part of the app, so the video would help find the source of the issue.

[Software-Design]

Having the same issue:

On a fresh install (using docker with the latest image available on docker hub, which is 2.6.11) I simply connected my Google Account and went to Settings > Calendar. It loads forever (since there are not cached events).

The request that keeps loading (and gets a timeout after 5 minutes) is /api/trpc/viewer.teams.getUpgradeable,viewer.teams.list,viewer.connectedCalendars?batch=1&input=%7B%220%22%3A%7B%22json%22%3Anull%2C%22meta%22%3A%7B%22values%22%3A%5B%22undefined%22%5D%7D%7D%2C%221%22%3A%7B%22json%22%3Anull%2C%22meta%22%3A%7B%22values%22%3A%5B%22undefined%22%5D%7D%7D%2C%222%22%3A%7B%22json%22%3Anull%2C%22meta%22%3A%7B%22values%22%3A%5B%22undefined%22%5D%7D%7D%7D

I can also see the error mentioned above in the docker logs:

@calcom/web:start: Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client @calcom/web:start: at new NodeError (node:internal/errors:387:5) @calcom/web:start: at ServerResponse.setHeader (node:_http_outgoing:644:11) @calcom/web:start: at ServerResponse._res.setHeader (/calcom/node_modules/next/dist/server/base-server.js:180:24) @calcom/web:start: at Function.getInitialProps (/calcom/apps/web/.next/server/pages/_document.js:126:22) @calcom/web:start: at Object. (/calcom/node_modules/next/dist/shared/lib/utils.js:84:33) @calcom/web:start: at Generator.next () @calcom/web:start: at asyncGeneratorStep (/calcom/node_modules/@swc/helpers/lib/_async_to_generator.js:13:28) @calcom/web:start: at _next (/calcom/node_modules/@swc/helpers/lib/_async_to_generator.js:31:17) @calcom/web:start: at /calcom/node_modules/@swc/helpers/lib/_async_to_generator.js:36:13 @calcom/web:start: at new Promise () @calcom/web:start: at Object. (/calcom/node_modules/@swc/helpers/lib/_async_to_generator.js:28:16) @calcom/web:start: at Object.loadGetInitialProps (/calcom/node_modules/next/dist/shared/lib/utils.js:62:33) @calcom/web:start: at loadDocumentInitialProps (/calcom/node_modules/next/dist/server/render.js:682:48) @calcom/web:start: at renderDocument (/calcom/node_modules/next/dist/server/render.js:732:45) @calcom/web:start: at /calcom/node_modules/next/dist/server/render.js:769:114 @calcom/web:start: at NextTracerImpl.trace (/calcom/node_modules/next/dist/server/lib/trace/tracer.js:56:20) { @calcom/web:start: code: 'ERR_HTTP_HEADERS_SENT' @calcom/web:start: }

EDIT: The (personal) settings page loads again after I disabled the Google Calendar integration as admin (saying that no calendar is linked). So the issue is definitely related to it. But I don't even get to the point that I have to active conflicts or anything.

[funsurfer2008]

I have tested it with the latest version it dont work at all brgds

[emrysal]

@funsurfer2008 Are you using Docker?

[funsurfer2008]

No. Local installiert.

[piec]

@piec when you have time please post the video regarding this. I am not seeing any doc related to this part of the app, so the video would help find the source of the issue.

Hi @laxmikanta415 here is a video of my test.

What's displayed:

• top left: my self hosted instance
• right: the calcom container logs
• bottom left: https://cal.com

My desktop environment uses focus on mouse move, which I assume triggers the visibilitychange event in the web page when I hover a browser window. It triggers an refresh of available times in the meeting page.

My self-hosted version has just been updated and is running the following image sha256:edce5f23880a49a1334a323095427744a11e3426d5db70fed51aefc9a9e6033f.

Hope it helps :)

calcom.webm

[laxmikanta415]

@piec when you have time please post the video regarding this. I am not seeing any doc related to this part of the app, so the video would help find the source of the issue.

Hi @laxmikanta415 here is a video of my test.

What's displayed:

• top left: my self hosted instance
• right: the calcom container logs
• bottom left: https://cal.com

My desktop environment uses focus on mouse move, which I assume triggers the visibilitychange event in the web page when I hover a browser window. It triggers an refresh of available times in the meeting page.

My self-hosted version has just been updated and is running the following image sha256:edce5f23880a49a1334a323095427744a11e3426d5db70fed51aefc9a9e6033f.

Hope it helps :)

Thank you for sharing the screencast, i completely understand the problem now. I will look into it.

[funsurfer2008]

perfect. the same problem is on sync with exchange account. So it is a Problem in general sync

brgds

[gabamnml]

same issue with self hosted in AWS. Events settings and google calendar dont update the data in frontend and in console get the error @calcom/web:start: Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client

[funsurfer2008]

Hy! is there a fix? this bug is a little showstopper for me ;)

brgds

[sypion]

I agree with @funsurfer2008! It's been happening for quite some time now and is making Cal unusable for both myself and those who I host it for.

[jacobzukerman]

Hi, we just updated our self-hosted integration from 2.5.5 to 2.7.15 and ran into the same issue. We have never experienced this problem before while on Cal for the past six or so months. We host around 100 tech leaders & coaches on our platform, and this has put us in a challenging position. Wish there was some big warming somewhere, which would have paused us from making the upgrade.

Can we please get an update? @PeerRich

[funsurfer2008]

@roae: fix is working if the second " is written right. for me it is working after rebuliding. brgds

[Tom-Potanski]

We experience the same issue.

[Software-Design]

On a fresh installed instance (using docker and v.2.8.9-sh-ce) this issue is still appearing.

[nickmonad]

For what it's worth, we had this same issue on 2.6.6. Upgraded to 2.9.6 and the issue seems to be resolved now.

[piec]

@nickmonad good to know, I'll try again :)

[roae]

This was fixed in this PR #8305, I going to close this, if the issue is still appearing, we can reopen it.

[piec]

I confirm it now works for me too, thanks :+1: