RuntimeError: There is no current event loop in thread 'raven-sentry.BackgroundWorker'

[lcannon]

Recently we've noticed that we're not getting sentry errors logged from our Django application. I can see the following traceback in our kubernetes logs on GCP:

  File "/usr/local/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/srv/venv/lib/python3.6/site-packages/sentry_sdk/integrations/threading.py", line 63, in run
    hub = parent_hub or Hub.current
  File "/srv/venv/lib/python3.6/site-packages/sentry_sdk/hub.py", line 112, in current
    rv = _local.get(None)
  File "/srv/venv/lib/python3.6/site-packages/aiocontextvars/var.py", line 33, in get
    ctx: Context = Context.current()
  File "/srv/venv/lib/python3.6/site-packages/aiocontextvars/context.py", line 26, in current
    local = Task.current_task() or _thread_local
  File "/usr/local/lib/python3.6/asyncio/events.py", line 694, in get_event_loop
    return get_event_loop_policy().get_event_loop()
  File "/usr/local/lib/python3.6/asyncio/events.py", line 602, in get_event_loop
    % threading.current_thread().name)
RuntimeError: There is no current event loop in thread 'raven-sentry.BackgroundWorker'.
"  

Unfortunately, since it's been broken for a while, I'm having trouble isolating exactly what changed to cause this to happen. We're using a fairly standard django application running Django 2.2.4 on uwsgi 2.0.18, with threading enabled, running inside a python:3.6.9-slim-stretch image.

We're using the same versions of the above to run containers with celery, which are working properly. I'm a bit stumped as to what the problem could be.

[lcannon]

This is with sentry-sdk==0.14.0 by the way

[untitaker]

Hi, is this the full stacktrace?

[untitaker]

@lcannon could you try removing aiocontextvars package from your environment? It seems like that one interferes with how thread locals work.

[lcannon]

Looks like that fixed the error -- it looks like the only reason we added this was because the sentry python page for django recommends adding it if you want channels (which we are also using) to have its reports properly instrumented, and are on python < 3.7. (https://docs.sentry.io/platforms/python/django/)

Does this mean if we want channels to be instrumented properly and avoid these errors we will need to update to python 3.7 or 3.8?

[untitaker]

@lcannon ah that is tricky. I think we can work around this issue in the SDK, but upgrading to Python 3.7 (and not using aiocontextvars) should fix it for sure.

[untitaker]

@lcannon so fwiw the "correct" way this would work under Python 3.6 is if aiocontextvars were available for processes that serve websocket/asgi requests, and not available for processes that serve basic wsgi. But that seems like bad UX. I am honestly not sure how to remediate this.

[lcannon]

Gotcha, I think I will just update our python version. Might be worth updating the docs to mention this?

Thanks for your help!

[untitaker]

We need to fix the sdk there, updating python is just a workaround

[untitaker]

actually @lcannon could you tell me more about the aiocontextvars you have installed? I am very suspicious because of this frame:

   File "/srv/venv/lib/python3.6/site-packages/aiocontextvars/var.py", line 33, in get

but the aiocontextvars I know consists of a single file aiocontextvars.py.

It seems like you're running a really old version of aiocontextvars. Please run

pip install -U aiocontextvars

[untitaker]

Right, this is broken with aiocontextvars==0.1.1, but works with newer versions:

from aiocontextvars import ContextVar
import threading

ctx = ContextVar("ctx")

def run():
    ctx.get("foo")
    ctx.set("bar")

t = threading.Thread(target=run)
t.start()
t.join()

[untitaker]

Closing, if somebody else is reading this, you need to update aiocontextvars package.