search

Crontap / crontap

Schedule recurring API calls without the hassle
https://crontap.com
11 stars 0 forks source link

Bug: Missed Cron Jobs this morning around 9am CST #9

Closed francistogram closed 8 months ago

francistogram commented 8 months ago

I've got a bunch of cron jobs that run every 5 minutes and this morning at 9am CST one was missed and looks like the other was delayed 4 minutes

Here is at 8:55am image

Next one is at 9:09:53am (4 minutes and 53 second late) image

And then the 3rd is at 9:10:51am (51 seconds late) image

The schedule ID is wqcSBfR07z4fsJP9CaqK but this happened across all of my jobs vIehCTtgKRhy6A9C2Tr8 this one too

Any idea what happened here @danmindru? This is a pretty serious issue

danmindru commented 8 months ago

hi @francistogram we’re investigating and will get back to you as soon as possible. We’ll treat this with outmost importance. Thank you for your patience.

danmindru commented 8 months ago

Hi @francistogram, First of all thank you for reporting the issue, the debug information was key to narrowing down the issue. We've analysed the incident and have concluded that for a short period all of Crontap instances (including redundant ones) where powered down at once. As they rebooted some jobs were locked from the previous instances, therefore some were delayed and/or started at odd times (when the lock expired). We place locks to prevent running jobs multiple times in such scenarios, however we concluded the current lock of 10 minutes can cause issues for schedules that are more frequent. We took a couple steps to prevent this:

Furthermore, we're in contact with Google to better understand how all instances powered down. We might consider a platform change depending on their assessment.

With the changes done today we hope the platform will be as reliable as it can be. However, with all infrastructure changes it's important to monitor the impact over the next few days & we'll be watching it closely. If you notice any anomalies after 4am CST please get in touch.

As a token of our appreciation for all the effort, we've changed your subscription to a free lifetime plan. You'll be able to always use Crontap for free ❤️

Thank you once again and all the best, Dan

francistogram commented 8 months ago

Hey @danmindru

Appreciate the quick resolution & all the context here! Hopefully Google is helpful in figuring out the initial cause

Will let you know if I see anything on my side going forwards!

As a token of our appreciation for all the effort, we've changed your subscription to a free lifetime plan. You'll be able to always use Crontap for free ❤️

This is amazing. You guys are the best. Shall keep you updated if I see anything here / keep throwing feature requests your way :)