Closed francistogram closed 8 months ago
hi @francistogram we’re investigating and will get back to you as soon as possible. We’ll treat this with outmost importance. Thank you for your patience.
Hi @francistogram, First of all thank you for reporting the issue, the debug information was key to narrowing down the issue. We've analysed the incident and have concluded that for a short period all of Crontap instances (including redundant ones) where powered down at once. As they rebooted some jobs were locked from the previous instances, therefore some were delayed and/or started at odd times (when the lock expired). We place locks to prevent running jobs multiple times in such scenarios, however we concluded the current lock of 10 minutes can cause issues for schedules that are more frequent. We took a couple steps to prevent this:
Furthermore, we're in contact with Google to better understand how all instances powered down. We might consider a platform change depending on their assessment.
With the changes done today we hope the platform will be as reliable as it can be. However, with all infrastructure changes it's important to monitor the impact over the next few days & we'll be watching it closely. If you notice any anomalies after 4am CST please get in touch.
As a token of our appreciation for all the effort, we've changed your subscription to a free lifetime plan. You'll be able to always use Crontap for free ❤️
Thank you once again and all the best, Dan
Hey @danmindru
Appreciate the quick resolution & all the context here! Hopefully Google is helpful in figuring out the initial cause
Will let you know if I see anything on my side going forwards!
As a token of our appreciation for all the effort, we've changed your subscription to a free lifetime plan. You'll be able to always use Crontap for free ❤️
This is amazing. You guys are the best. Shall keep you updated if I see anything here / keep throwing feature requests your way :)
I've got a bunch of cron jobs that run every 5 minutes and this morning at 9am CST one was missed and looks like the other was delayed 4 minutes
Here is at 8:55am
Next one is at 9:09:53am (4 minutes and 53 second late)
And then the 3rd is at 9:10:51am (51 seconds late)
The schedule ID is
wqcSBfR07z4fsJP9CaqK
but this happened across all of my jobsvIehCTtgKRhy6A9C2Tr8
this one tooAny idea what happened here @danmindru? This is a pretty serious issue