search

getsentry / sentry-python

The official Python SDK for Sentry.io
https://sentry.io/for/python/
MIT License
1.93k stars 509 forks source link

Sentry and serverless sdk not working well together #2948

Closed zacroker closed 7 months ago

zacroker commented 7 months ago

How do you use Sentry?

Sentry Saas (sentry.io)

Version

1.40.5

Steps to Reproduce

Unfortunately this error is sporadic and we are not able to consistently reproduce it, though we can see it is happening from the sentry dashboard.

POTENTIALLY RELATED TO THIS TICKET

  1. We have a python django application running in lambda (v3.9)
  2. We have sentry configured using sentry-python in our initialisation code
  3. We have the python requirements in a lambda layer (serverless-python-requirements)
  4. Problem:
    • Occasionally when making API requests we get unexplained 500 errors
    • Occasionally, some of our data pipeline functions crash
  5. There is a common set of log lines when these crashes occur:
    2024-04-07T22:09:33.058+10:00   Exception in thread sentry.profiler.ThreadScheduler:
    2024-04-07T22:09:33.058+10:00   Traceback (most recent call last):
    2024-04-07T22:09:33.058+10:00   File "/var/lang/lib/python3.9/threading.py", line 980, in _bootstrap_inner
    2024-04-07T22:09:33.059+10:00   self.run()
    2024-04-07T22:09:33.059+10:00   File "/opt/python/sentry_sdk/integrations/threading.py", line 72, in run
    2024-04-07T22:09:33.059+10:00   reraise(*_capture_exception())
    2024-04-07T22:09:33.059+10:00   File "/opt/python/sentry_sdk/_compat.py", line 127, in reraise
    2024-04-07T22:09:33.059+10:00   raise value
    2024-04-07T22:09:33.059+10:00   File "/opt/python/sentry_sdk/integrations/threading.py", line 70, in run
    2024-04-07T22:09:33.059+10:00   return old_run_func(self, *a, **kw)
    2024-04-07T22:09:33.059+10:00   File "/var/lang/lib/python3.9/threading.py", line 917, in run
    2024-04-07T22:09:33.059+10:00   self._target(*self._args, **self._kwargs)
    2024-04-07T22:09:33.059+10:00   File "/opt/python/sentry_sdk/profiler.py", line 943, in run
    2024-04-07T22:09:33.059+10:00   self.sampler()
    2024-04-07T22:09:33.059+10:00   File "/opt/python/sentry_sdk/profiler.py", line 828, in _sample_stack
    2024-04-07T22:09:33.059+10:00   sample = [
    2024-04-07T22:09:33.059+10:00   File "/opt/python/sentry_sdk/profiler.py", line 829, in <listcomp>
    2024-04-07T22:09:33.060+10:00   (str(tid), extract_stack(frame, cache, cwd))
    2024-04-07T22:09:33.060+10:00   File "/opt/python/sentry_sdk/profiler.py", line 274, in extract_stack
    2024-04-07T22:09:33.060+10:00   frame_ids = tuple(frame_id(raw_frame) for raw_frame in raw_frames)
    2024-04-07T22:09:33.060+10:00   File "/opt/python/sentry_sdk/profiler.py", line 274, in <genexpr>
    2024-04-07T22:09:33.060+10:00   frame_ids = tuple(frame_id(raw_frame) for raw_frame in raw_frames)
    2024-04-07T22:09:33.060+10:00   File "/opt/python/sentry_sdk/profiler.py", line 301, in frame_id
    2024-04-07T22:09:33.060+10:00   return (raw_frame.f_code.co_filename, raw_frame.f_lineno, get_frame_name(raw_frame))
    2024-04-07T22:09:33.060+10:00   File "/opt/python/sentry_sdk/profiler.py", line 361, in get_frame_name
    2024-04-07T22:09:33.060+10:00   for cls in frame.f_locals["self"].__class__.__mro__:
    2024-04-07T22:09:33.060+10:00   File "/var/task/serverless_sdk/vendor/wrapt/wrappers.py", line 100, in __class__
    2024-04-07T22:09:33.060+10:00   return self.__wrapped__.__class__
    2024-04-07T22:09:33.060+10:00   File "/var/task/serverless_sdk/vendor/wrapt/wrappers.py", line 194, in __getattr__
    2024-04-07T22:09:33.060+10:00   raise ValueError('wrapper has not been initialised')
    2024-04-07T22:09:33.060+10:00   ValueError: wrapper has not been initialised

Any advice appreciated!

Expected Result

Don't crash 😅

Actual Result

Crash

sentrivana commented 7 months ago

Hey @zacroker, thanks for writing in! 👋🏻

From looking at the logs you posted and some serverless_sdk/wrapt code I don't believe this is related to the other issue where the SDK is unable to start a thread -- in your case the thread is already running.

What you're seeing is a profiler issue. It's trying to inspect something that's not supposed to be inspected and hitting this.

The SDK needs to be smarter here and guard against possible exceptions. We'll fix this. In the meantime, disabling the profiler should get rid of the errors.