search

getsentry / sentry-python

The official Python SDK for Sentry.io
https://sentry.io/for/python/
MIT License
1.86k stars 483 forks source link

Celery Cron: Spurious missed_checkin issues #3179

Open rainsville opened 2 months ago

rainsville commented 2 months ago

How do you use Sentry?

Self-hosted/on-premise

Version

2.5.1

Steps to Reproduce

We have a Celery Redbeat setup with tasks that are scheduled to run every minute (using the interval scheduler). In the Sentry UI, we can see that Sentry expects the tasks to run at hh:mm:00. In practice, the tasks are started on time, but around the hh:mm:57 mark.

Occasionally, Sentry will report tasks as missing. We have noticed two different scenarios:

Expected Result

The expectation is to not mark the tasks as missed. To account for potential delays in reporting the checkin to Sentry, it may be useful to allow specifying a checkin_margin for Celery tasks.

Actual Result

Example 1: task is expected to start at Jun 18, 12:27:00 PM UTC, is reported as started at Jun 18, 12:28:02 PM UTC and marked as Missed. In the Celery logs, the task is actually started at Jun 18, 12:27:57 PM UTC.

image

Example 2: task is expected to start at Jun 18, 12:37:00 PM UTC, starts at Jun 18, 12:37:56 PM UTC, but is still marked as Missed.

image

sentrivana commented 2 months ago

Hey @rainsville, thanks for writing in.

Could you add debug=True to your sentry_sdk.init and check whether you see an event being sent out for the missed check-ins? Trying to determine whether this is an issue between the SDK and Sentry or if the check-ins are not even being emitted by the SDK.

I'd also recommend you try keep_alive=True. This seemed to improve the situation for folks who reported similar behavior, so it might also help in your case.