Got RecursionError during any request that requires DB access.
The service is currently configured to have the following related components enabled:
Django Debug Toolbar
Django Silk
Django Cachalot
Django Redis
Looking at the stack trace it seems get_many was called 890 times, causing Python to exceed the 1000 recursion limit.
Cronjob tasks triggered by Django Celery Beats do not have this issue. Currently, it appears only external HTTP requests have this issue. One suspicion is some kind of conflict with Django Debug Toolbar's cache panel.
Stack trace:
Notice in the middle it said : [Previous line repeated 896 more times]
2024-08-26 10:38:12,840 <9> [ERROR] Internal Server Error: /index/
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
response = get_response(request)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/django/middleware.py", line 175, in __call__
return f(*args, **kwargs)
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/silk/middleware.py", line 70, in __call__
self.process_request(request)
File "/usr/local/lib/python3.11/site-packages/silk/profiling/profiler.py", line 57, in wrapped_target
result = target(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/silk/middleware.py", line 104, in process_request
if not _should_intercept(request):
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/silk/middleware.py", line 41, in _should_intercept
if not config.SILKY_INTERCEPT_FUNC(request):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/central/central/core/utils/functional.py", line 24, in __call__
return self.func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/central/central/core/monitor/silk.py", line 11, in silk_should_intercept
decision = DebugRule.decide(method, full_path, "silk")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/central/central/apps/base/models/debug.py", line 98, in decide
rules = list(DebugRule.objects.all())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 400, in __iter__
self._fetch_all()
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 1928, in _fetch_all
self._result_cache = list(self._iterable_class(self))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 91, in __iter__
results = compiler.execute_sql(
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/silk/sql.py", line 100, in execute_sql
return self._execute_sql(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/cachalot/monkey_patch.py", line 37, in inner
return original(compiler, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/cachalot/monkey_patch.py", line 96, in inner
return _get_result_or_execute_query(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/cachalot/monkey_patch.py", line 46, in _get_result_or_execute_query
data = cache.get_many(table_cache_keys + [cache_key])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/cachalot/transaction.py", line 18, in get_many
data.update(self.parent_cache.get_many(missing_keys))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/cachalot/transaction.py", line 18, in get_many
data.update(self.parent_cache.get_many(missing_keys))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/cachalot/transaction.py", line 18, in get_many
data.update(self.parent_cache.get_many(missing_keys))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[Previous line repeated 896 more times] ### !!! <---- Notice this line indicating where infinite recursion happens
File "/usr/local/lib/python3.11/site-packages/debug_toolbar/panels/cache.py", line 40, in wrapper
return original_method(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django_redis/cache.py", line 29, in _decorator
return method(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django_redis/cache.py", line 122, in get_many
return self.client.get_many(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django_redis/client/default.py", line 485, in get_many
results = client.mget(*map_keys)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/redis/commands/core.py", line 2015, in mget
return self.execute_command("MGET", *args, **options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/sentry_sdk/integrations/redis/__init__.py", line 231, in sentry_patched_execute_command
with hub.start_span(op=OP.DB_REDIS, description=description) as span:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/sentry_sdk/hub.py", line 448, in start_span
return scope.start_span(span=span, instrumenter=instrumenter, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/sentry_sdk/scope.py", line 789, in start_span
new_child_span = active_span.start_child(**kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/sentry_sdk/tracing.py", line 255, in start_child
child = Span(
^^^^^
File "/usr/local/lib/python3.11/site-packages/sentry_sdk/tracing.py", line 146, in __init__
self.span_id = span_id or uuid.uuid4().hex[16:]
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/uuid.py", line 723, in uuid4
return UUID(bytes=os.urandom(16), version=4)
^^^^^^^^^^^^^^
RecursionError: maximum recursion depth exceeded while calling a Python object
What happened?
Got RecursionError during any request that requires DB access. The service is currently configured to have the following related components enabled:
Looking at the stack trace it seems get_many was called 890 times, causing Python to exceed the 1000 recursion limit. Cronjob tasks triggered by Django Celery Beats do not have this issue. Currently, it appears only external HTTP requests have this issue. One suspicion is some kind of conflict with Django Debug Toolbar's cache panel.
Steps to reproduce
Django 5.0.5 Django Debug Toolbar 4.2.0 Django Silk 5.1.0 Django Cachalot 2.6.2 Django Redis 5.4.0
Stack trace: Notice in the middle it said :
[Previous line repeated 896 more times]