search

getsentry / raven-python

Raven is the legacy Python client for Sentry (getsentry.com) — replaced by sentry-python
https://sentry.io
BSD 3-Clause "New" or "Revised" License
1.68k stars 657 forks source link

Fix celery error logging #1267

Open tilboerner opened 6 years ago

tilboerner commented 6 years ago

Error logging from Celery tasks is broken out of the box. While uncaught exception reporting works, error-level log messages from inside Celery worker do not get reported to Sentry.

For the user, it's not obvious that this is happening, or what can be done to fix it.The setup instructions for Django and Celery are not sufficient to get it working. There is documentation on raven.readthedocs.io suggesting that the Django handler "captures the errors" from workers. Consequently, there is #922, suggesting a workaround.

The problem is that when setting up logging for worker processes, Celery configures the task logger to not propagate to the root logger:

https://github.com/celery/celery/blob/47ca2b462f22a8d48ed8d80c2f9bf8b9dc4a4de6/celery/app/log.py#L132

https://github.com/celery/celery/blob/47ca2b462f22a8d48ed8d80c2f9bf8b9dc4a4de6/celery/app/log.py#L176

This prevents log events from reaching the Celery handler in the root logger. (Note that this setup does not happen in the Django shell, for example. There, things work fine.)

The workaround in #922 disables this logging setup completely. I'd argue that it's cleaner to work with whatever Celery decides to do and not mess with the logging system any further, beyond adding sentry handlers.

So, this PR adds a signal handler for the task logger, to give it the sentry handler if needed. It also adds code so that the DjangoCeleryHandler sets this up properly out of the box, if any of the CELERY_LOGLEVEL settings are defined. (This way it's opt-in; it could also be enabled by default.)

People should not have to discover the hard way that their errors do not properly get logged, so I think Raven should really fix this and work out of the box. If any manual action remains to be required, this should be noted in the documentation.

If you're willing to consider this PR, I'd be happy to add tests and documentation. In that case, it would be great if you could point me towards the proper places to put them.

ashwoods commented 6 years ago

@tilboerner thx! Mind rebasing with master to get the tests passing.

tilboerner commented 6 years ago

@ashwoods Rebased it. Could you restart that one failed travis job? Looks like some pip download timed out.

blodone commented 6 years ago

its really really painful to enable logging celery to sentry... +1

huangsam commented 6 years ago

Is there anything blocking this from getting released? This PR would be incredibly useful for folks who need to track Celery errors via the Raven client.

untitaker commented 6 years ago

To all, I think the proper fix is to upgrade to the new Python SDK which bypasses those problems entirely.

I'm a bit vary of fixing this in raven as it is a breaking change for something that is effectively in maintenance mode.