search

Netflix-Skunkworks / raven-python-lambda

Sentry/Raven SDK Integration For AWS Lambda (python) and Serverless
Apache License 2.0
47 stars 15 forks source link

RuntimeError: can't start new thread in AWS Lambda #38

Closed ranrib closed 6 years ago

ranrib commented 6 years ago

From time to time, we get this error:

File "/var/task/handlers/main.py.py", line 19, in handler
    return ''
  File "/var/task/raven_python_lambda/__init__.py", line 208, in decorated
    raise e
  File "/var/task/raven_python_lambda/__init__.py", line 202, in decorated
    timers = install_timers(self.config, context)
  File "/var/task/raven_python_lambda/__init__.py", line 272, in install_timers
    t.start()
  File "/var/lang/lib/python3.6/threading.py", line 846, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
mikegrima commented 6 years ago

These are usually transient errors that happen with AWS Lambda from time to time. Your function will re-start after this and it should work again.

It does create some noisy log files. Probably worth following up with AWS on why that happens every now and then.

lbalceda commented 5 years ago

@ranrib are you still seeing this? @mikegrima my team frequently sees this whenever we try to do ~500 invocations at once. For now will try setting SENTRY_CAPTURE_MEMORY=FALSE and SENTRY_CAPTURE_TIMEOUTS=FALSE so it doesn't try to add any timers..

image

image

mikegrima commented 5 years ago

@lbalceda : I would definitely share that with AWS. I do know that the underlying instance that runs the lambdas keeps memory warm for future lambda invocations.

ranrib commented 5 years ago

@lbalceda we haven't managed to fix that. We disabled it like you did. Ultimately we moved to use our own wrapper that got this feature: https://github.com/epsagon/epsagon-python