Calendar feed updating really slowly

[DarkFox]

I've missed the start of a few runs due to the calendar feed updating very slowly.

As I understand it, you're running the calendar through Google Calendar, which is likely causing the delay.

Have you considered running the feed through a free CDN like Cloudflare? This would protect your server while providing faster updates for everyone.

[Aldaviva]

Hi @DarkFox,

I know what you mean, the Google Calendar Android app on my phone is always notifying me of runs late.

The current situation is a little bit different. The calendar feed is actually self-hosted rather than hosted on Google Calendar, and my server updates very quickly: right now the max update frequency is every 3 minutes (only to avoid hammering GDQ's servers).

The slowdown comes when someone uses Google Calendar as a client to subscribe to this feed, because Google refuses to poll my server for updates any faster than once every few hours. Therefore, the bottleneck is on Google's side, not mine, so I can't control it, not even with a Cache-Control: max-age response header. Also, iCalendar sadly doesn't have a push notification system like RSS's rssCloud webhook eventing.

Adding a CDN would reduce load on my server in the face of very high request volume, but at this point it's not high enough to stress my server with its simple in-memory caching. It could also improve latency across regions due to faster upload speeds, but the payloads are relatively small, and ICS requests are generally not latency-sensitive due to being fetched in the background. Also, CDNs generally work best with longer cache durations, so I've avoided using one here. Overall, adding a CDN would just result in Google polling the CDN slowly, and the problem would not be solved.

Since I can't think of a way to convince Google Calendar to poll my server more frequently, the only technique I can offer is using a different iCalendar client. I personally use Vivaldi's calendar system to subscribe to this calendar, for example.

[Aldaviva]

Since this seems to be the most frequently-encountered issue with this project, and since Google Calendar is so popular, here is an idea. I'm not sure if this is a good idea or not, but I might try to find the time to add a new feature to this project to automatically mirror these events into a separate calendar hosted in Google Calendar, which would ideally update faster than the external ICS file that Google polls slowly, because it would be a push instead of a poll/pull. In this scenario, users would subscribe to a public calendar shared by another Google Calendar user, not to a URL like they do today. Hopefully it can at least be stateless, for my sanity.

So far I only have auth and reads working, it still needs other functionality like variable timers, event identity, event diffing, and writes.

[DarkFox]

I've been pulling the feed directly into Home Assistant using the ics_calendar integration, but there still seemed to be some delay.

For now I've dockerized the executable and am running it locally, which seems to work a little better. Another issue might also simply be how quickly the GDQ team updates the schedule.

[Aldaviva]

Added a new integration that pushes events to Google Calendar every minute to avoid Google's slow polling of external ICS files. This should make updates appear much faster for Google Calendar users.