search

elastic / apm-agent-python

https://www.elastic.co/guide/en/apm/agent/python/current/index.html
BSD 3-Clause "New" or "Revised" License
408 stars 213 forks source link

Django Project integration #691

Closed vclaudic closed 4 years ago

vclaudic commented 4 years ago

Hi,

First of all, please receive my best wishes for these new Year and decade for you and your relatives ! :-)

Yet, let's talk about my mistake. I've started, few weeks ago, a new Django project based on Django 2.2.9, Python 3.7.6, PostgreSQL 12.1, Apache 2.4.41, mod_wsgi 4.7 and the Django modules elasticsearch 7.1, elasticsearch-dsl 7.1 and elastic_apm 5.3.3.

With "'elasticapm.contrib.django'" declared in installed-apps, as long as DEBUG is set to True, I've no mistake, i.e. every thing works fine, but as soon as I'm setting it to False, both Apache or the runserver are sadly crashing. Here is a short extract from Apache error_log file: ——— … [Mon Jan 13 11:56:39.486822 2020] [core:notice] [pid 21292:tid 139720174155584] AH00052: child pid 21298 exit signal Aborted (6) [Mon Jan 13 11:56:39.662442 2020] [mpm_event:notice] [pid 22765:tid 139947965765440] AH00489: Apache/2.4.41 (Unix) OpenSSL/1.1.1d mod_wsgi/4.7.0 Python/3.7 configured -- resuming normal operations [Mon Jan 13 11:56:39.662634 2020] [core:notice] [pid 22765:tid 139947965765440] AH00094: Command line: '/ecq/run/apache/bin/httpd' [Tue Jan 14 11:43:55.293070 2020] [mpm_event:notice] [pid 22765:tid 139947965765440] AH00492: caught SIGWINCH, shutting down gracefully Error in `/ecq/run/apache/bin/httpd': double free or corruption (out): 0x00007f47d400c460 ======= Backtrace: ========= /lib64/libc.so.6(+0x81679)[0x7f482b62f679] /ecq/run/lib/libpython3.7m.so.1.0(PyThreadState_DeleteCurrent+0x1e)[0x7f48257f3a0e] /ecq/run/lib/libpython3.7m.so.1.0(+0x24a472)[0x7f482580c472] /ecq/run/lib/libpython3.7m.so.1.0(+0x236224)[0x7f48257f8224] /lib64/libpthread.so.0(+0x7e65)[0x7f482bb87e65] /lib64/libc.so.6(clone+0x6d)[0x7f482b6ac88d] … ———

The mistake was the same with the 5.3.2 version of APM and here are some additional trace: ——— … [Tue Jan 14 15:01:39.460247 2020] [core:notice] [pid 20504:tid 140590202570560] AH00052: child pid 24941 exit signal Aborted (6) [Tue Jan 14 15:01:39.460391 2020] [core:notice] [pid 20504:tid 140590202570560] AH00052: child pid 20866 exit signal Aborted (6) [Tue Jan 14 15:01:39.628508 2020] [mpm_event:notice] [pid 27255:tid 140545858905920] AH00489: Apache/2.4.41 (Unix) OpenSSL/1.1.1d mod_wsgi/4.7.0 Python/3.7 configured -- resuming normal operations [Tue Jan 14 15:01:39.628688 2020] [core:notice] [pid 27255:tid 140545858905920] AH00094: Command line: '/ecq/run/apache/bin/httpd' Fatal Python error: Cannot recover from stack overflow.

Thread 0x00007fd361fea740 (most recent call first): [Tue Jan 14 15:01:57.663265 2020] [core:notice] [pid 27255:tid 140545858905920] AH00052: child pid 27259 exit signal Aborted (6) Fatal Python error: Cannot recover from stack overflow.

Thread 0x00007fd361fea740 (most recent call first): [Tue Jan 14 15:02:00.675209 2020] [core:notice] [pid 27255:tid 140545858905920] AH00052: child pid 27261 exit signal Aborted (6) Fatal Python error: Cannot recover from stack overflow.

Current thread 0x00007fd34dffb700 (most recent call first): File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/init.py", line 58 in varmap File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/base.py", line 177 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/stacks.py", line 299 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/stacks.py", line 299 in get_frame_info File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/contrib/django/utils.py", line 84 in iterate_with_template_sources File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/contrib/django/client.py", line 207 in _get_stack_info_for_trace File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/base.py", line 170 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 432 in end File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 275 in end_span File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 599 in exit File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/instrumentation/packages/psycopg2.py", line 87 in call File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/instrumentation/packages/dbapi2.py", line 257 in call_if_sampling File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/postgresql/base.py", line 178 in get_new_connection File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 195 in connect File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 217 in ensure_connection File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 233 in _cursor File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 256 in cursor File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/sql/compiler.py", line 1131 in execute_sql File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 55 in iter File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 1242 in _fetch_all File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 274 in iter File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 250 in repr File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/encoding.py", line 151 in transform File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/encoding.py", line 205 in shorten File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/base.py", line 175 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/init.py", line 68 in varmap File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/base.py", line 177 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/stacks.py", line 299 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/stacks.py", line 299 in get_frame_info File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/contrib/django/utils.py", line 84 in iterate_with_template_sources File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/contrib/django/client.py", line 207 in _get_stack_info_for_trace File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/base.py", line 170 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 432 in end File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 275 in end_span File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 599 in exit File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/instrumentation/packages/psycopg2.py", line 87 in call File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/instrumentation/packages/dbapi2.py", line 257 in call_if_sampling File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/postgresql/base.py", line 178 in get_new_connection File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 195 in connect File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 217 in ensure_connection File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 233 in _cursor File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 256 in cursor File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/sql/compiler.py", line 1131 in execute_sql File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 55 in iter File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 1242 in _fetch_all File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 274 in iter File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 250 in repr File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/encoding.py", line 151 in transform File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/encoding.py", line 205 in shorten File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/base.py", line 175 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/init.py", line 68 in varmap File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/base.py", line 177 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/stacks.py", line 299 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/stacks.py", line 299 in get_frame_info File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/contrib/django/utils.py", line 84 in iterate_with_template_sources File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/contrib/django/client.py", line 207 in _get_stack_info_for_trace File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/base.py", line 170 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 432 in end File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 275 in end_span File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 599 in exit File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/instrumentation/packages/psycopg2.py", line 87 in call File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/instrumentation/packages/dbapi2.py", line 257 in call_if_sampling File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/postgresql/base.py", line 178 in get_new_connection File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 195 in connect File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 217 in ensure_connection File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 233 in _cursor File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 256 in cursor File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/sql/compiler.py", line 1131 in execute_sql File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 55 in iter File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 1242 in _fetch_all File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 274 in iter File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 250 in repr File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/encoding.py", line 151 in transform File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/encoding.py", line 205 in shorten File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/base.py", line 175 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/init.py", line 68 in varmap File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/base.py", line 177 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/stacks.py", line 299 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/stacks.py", line 299 in get_frame_info File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/contrib/django/utils.py", line 84 in iterate_with_template_sources File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/contrib/django/client.py", line 207 in _get_stack_info_for_trace File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/base.py", line 170 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 432 in end File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 275 in end_span File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 599 in exit File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/instrumentation/packages/psycopg2.py", line 87 in call File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/instrumentation/packages/dbapi2.py", line 257 in call_if_sampling File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/postgresql/base.py", line 178 in get_new_connection File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 195 in connect File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 217 in ensure_connection File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 233 in _cursor File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 256 in cursor File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/sql/compiler.py", line 1131 in execute_sql File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 55 in iter File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 1242 in _fetch_all File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 274 in iter File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 250 in repr File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/encoding.py", line 151 in transform File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/encoding.py", line 205 in shorten File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/base.py", line 175 in ...

Thread 0x00007fd355ca4700 (most recent call first):

Thread 0x00007fd3477ee700 (most recent call first):

Thread 0x00007fd347fef700 (most recent call first):

Thread 0x00007fd349ff3700 (most recent call first):

Thread 0x00007fd30a09f700 (most recent call first): File "/ecq/run/lib/python3.7/threading.py", line 300 in wait File "/ecq/run/lib/python3.7/threading.py", line 552 in wait File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/threading.py", line 74 in run File "/ecq/run/lib/python3.7/threading.py", line 926 in _bootstrap_inner File "/ecq/run/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x00007fd30a8a0700 (most recent call first): File "/ecq/run/lib/python3.7/threading.py", line 300 in wait File "/ecq/run/lib/python3.7/threading.py", line 552 in wait File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/threading.py", line 74 in run File "/ecq/run/lib/python3.7/threading.py", line 926 in _bootstrap_inner File "/ecq/run/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x00007fd30b0a1700 (most recent call first): File "/ecq/run/lib/python3.7/threading.py", line 300 in wait File "/ecq/run/lib/python3.7/queue.py", line 179 in get File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/transport/base.py", line 126 in _process_queue File "/ecq/run/lib/python3.7/threading.py", line 870 in run File "/ecq/run/lib/python3.7/threading.py", line 926 in _bootstrap_inner File "/ecq/run/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x00007fd34a7f4700 (most recent call first): [Tue Jan 14 15:04:04.937796 2020] [core:notice] [pid 27255:tid 140545858905920] AH00052: child pid 27378 exit signal Aborted (6) Fatal Python error: Cannot recover from stack overflow.

Current thread 0x00007fd354ca2700 (most recent call first): File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/init.py", line 58 in varmap File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/base.py", line 177 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/stacks.py", line 299 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/stacks.py", line 299 in get_frame_info File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/contrib/django/utils.py", line 84 in iterate_with_template_sources File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/contrib/django/client.py", line 207 in _get_stack_info_for_trace File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/base.py", line 170 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 432 in end File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 275 in end_span File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 599 in exit File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/instrumentation/packages/psycopg2.py", line 87 in call File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/instrumentation/packages/dbapi2.py", line 257 in call_if_sampling File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/postgresql/base.py", line 178 in get_new_connection File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 195 in connect File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 217 in ensure_connection File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 233 in _cursor File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 256 in cursor File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/sql/compiler.py", line 1131 in execute_sql File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 55 in iter File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 1242 in _fetch_all File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 274 in iter File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 250 in repr File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/encoding.py", line 151 in transform File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/encoding.py", line 205 in shorten File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/base.py", line 175 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/init.py", line 68 in varmap File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/base.py", line 177 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/stacks.py", line 299 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/stacks.py", line 299 in get_frame_info File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/contrib/django/utils.py", line 84 in iterate_with_template_sources File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/contrib/django/client.py", line 207 in _get_stack_info_for_trace File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/base.py", line 170 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 432 in end File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 275 in end_span File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 599 in exit File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/instrumentation/packages/psycopg2.py", line 87 in call File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/instrumentation/packages/dbapi2.py", line 257 in call_if_sampling File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/postgresql/base.py", line 178 in get_new_connection File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 195 in connect File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 217 in ensure_connection File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 233 in _cursor File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 256 in cursor File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/sql/compiler.py", line 1131 in execute_sql File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 55 in iter File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 1242 in _fetch_all File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 274 in iter File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 250 in repr File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/encoding.py", line 151 in transform File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/encoding.py", line 205 in shorten File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/base.py", line 175 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/init.py", line 68 in varmap File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/base.py", line 177 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/stacks.py", line 299 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/stacks.py", line 299 in get_frame_info File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/contrib/django/utils.py", line 84 in iterate_with_template_sources File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/contrib/django/client.py", line 207 in _get_stack_info_for_trace File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/base.py", line 170 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 432 in end File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 275 in end_span File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 599 in exit File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/instrumentation/packages/psycopg2.py", line 87 in call File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/instrumentation/packages/dbapi2.py", line 257 in call_if_sampling File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/postgresql/base.py", line 178 in get_new_connection File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 195 in connect File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 217 in ensure_connection File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 233 in _cursor File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 256 in cursor File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/sql/compiler.py", line 1131 in execute_sql File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 55 in iter File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 1242 in _fetch_all File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 274 in iter File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 250 in repr File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/encoding.py", line 151 in transform File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/encoding.py", line 205 in shorten File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/base.py", line 175 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/init.py", line 68 in varmap File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/base.py", line 177 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/stacks.py", line 299 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/stacks.py", line 299 in get_frame_info File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/contrib/django/utils.py", line 84 in iterate_with_template_sources File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/contrib/django/client.py", line 207 in _get_stack_info_for_trace File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/base.py", line 170 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 432 in end File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 275 in end_span File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 599 in exit File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/instrumentation/packages/psycopg2.py", line 87 in call File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/instrumentation/packages/dbapi2.py", line 257 in call_if_sampling File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/postgresql/base.py", line 178 in get_new_connection File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 195 in connect File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 217 in ensure_connection File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 233 in _cursor File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 256 in cursor File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/sql/compiler.py", line 1131 in execute_sql File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 55 in iter File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 1242 in _fetch_all File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 274 in iter File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 250 in repr File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/encoding.py", line 151 in transform File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/encoding.py", line 205 in shorten File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/base.py", line 175 in ...

Thread 0x00007fd3574a7700 (most recent call first):

Thread 0x00007fd3564a5700 (most recent call first):

Thread 0x00007fd33a09f700 (most recent call first): File "/ecq/run/lib/python3.7/threading.py", line 300 in wait File "/ecq/run/lib/python3.7/threading.py", line 552 in wait File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/threading.py", line 74 in run File "/ecq/run/lib/python3.7/threading.py", line 926 in _bootstrap_inner File "/ecq/run/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x00007fd33a8a0700 (most recent call first): File "/ecq/run/lib/python3.7/threading.py", line 300 in wait File "/ecq/run/lib/python3.7/threading.py", line 552 in wait File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/utils/threading.py", line 74 in run File "/ecq/run/lib/python3.7/threading.py", line 926 in _bootstrap_inner File "/ecq/run/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x00007fd33b0a1700 (most recent call first): File "/ecq/run/lib/python3.7/threading.py", line 300 in wait File "/ecq/run/lib/python3.7/queue.py", line 179 in get File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.2-py3.7-linux-x86_64.egg/elasticapm/transport/base.py", line 126 in _process_queue File "/ecq/run/lib/python3.7/threading.py", line 870 in run File "/ecq/run/lib/python3.7/threading.py", line 926 in _bootstrap_inner File "/ecq/run/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x00007fd356ca6700 (most recent call first): [Tue Jan 14 15:10:06.693626 2020] [core:notice] [pid 27255:tid 140545858905920] AH00052: child pid 27260 exit signal Aborted (6) Fatal Python error: Cannot recover from stack overflow.

Thread 0x00007fd361fea740 (most recent call first): [Tue Jan 14 15:13:34.142416 2020] [core:notice] [pid 27255:tid 140545858905920] AH00052: child pid 28067 exit signal Aborted (6) Fatal Python error: Cannot recover from stack overflow.

Thread 0x00007fd361fea740 (most recent call first): [Tue Jan 14 15:13:37.147387 2020] [core:notice] [pid 27255:tid 140545858905920] AH00052: child pid 28960 exit signal Aborted (6) Fatal Python error: Cannot recover from stack overflow.

Thread 0x00007fd361fea740 (most recent call first): [Tue Jan 14 15:35:19.435248 2020] [core:notice] [pid 27255:tid 140545858905920] AH00052: child pid 29382 exit signal Aborted (6) Fatal Python error: Cannot recover from stack overflow.

Thread 0x00007fd361fea740 (most recent call first): [Tue Jan 14 15:35:22.438910 2020] [core:notice] [pid 27255:tid 140545858905920] AH00052: child pid 27463 exit signal Aborted (6) … ———

Regarding the mod_wsgi apache module, here is how it's called in Apache configuration: ——— … WSGIDaemonProcess calliope.ecqualis.lan processes=2 threads=15 display-name=%{GROUP} … ———

I've the same mistake when increasing the processes to 8 and threads to 25.

Here is the way Apache was built: ——— …

/ecq/run/apache/bin/httpd -V

Server version: Apache/2.4.41 (Unix) Server built: Nov 21 2019 08:11:04 Server's Module Magic Number: 20120211:88 Server loaded: APR 1.7.0, APR-UTIL 1.6.1 Compiled using: APR 1.7.0, APR-UTIL 1.6.1 Architecture: 64-bit Server MPM: event threaded: yes (fixed thread count) forked: yes (variable process count) Server compiled with.... -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) -D APR_USE_PROC_PTHREAD_SERIALIZE -D APR_USE_PTHREAD_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D DYNAMIC_MODULE_LIMIT=256 -D HTTPD_ROOT="/ecq/run/apache" -D SUEXEC_BIN="/ecq/run/apache/bin/suexec" -D DEFAULT_PIDLOG="logs/httpd.pid" -D DEFAULT_SCOREBOARD="logs/apache_runtime_status" -D DEFAULT_ERRORLOG="logs/error_log" -D AP_TYPES_CONFIG_FILE="conf/mime.types" -D SERVER_CONFIG_FILE="conf/httpd.conf"

… ———

As I've deployed an internal ELK instance (7.5.1 versions) for that project, I'll be pleased to also use it with APM.

So for any additional information allowing you to provide some help, please just drop me a line.

I thank you in advance for the time you'll spent on this and wish you a good day.

All the best,

Vincent

beniwohli commented 4 years ago

Hi @vclaudic

thanks for the extensive bug report! Do you see the same stack traces when running under mod_wsgi and with runserver? If it's only with mod_wsgi, you could try and add 

WSGIApplicationGroup %{GLOBAL}

to your mod_wsgi configuration. From the mod_wsgi docs:

When this is used with the %{GLOBAL} value, it forces the WSGI application to run in the main Python interpreter context of each process. This is preferred in this scenario as some third party packages for Python which include C extensions will not run in the Python sub interpreter contexts which mod_wsgi would use by default. By using the main Python interpreter context you eliminate the possibility of such third party packages for Python causing problems.

vclaudic commented 4 years ago

Hi Benjamin,

Thank you for your answer. Here are my answer in regard:

-> When running with runserver, here is what’s happen when trying to log in the admin GUI:

——— … [/ecq/run/repositories/calliope]# python3 manage.py runserver Performing system checks...

System check identified no issues (0 silenced). January 16, 2020 - 11:05:20 Django version 2.2.9, using settings 'calliope.settings' Starting development server at http://127.0.0.1:8000/ Quit the server with CONTROL-C. [16/Jan/2020 11:05:33] "GET / HTTP/1.1" 404 77 [16/Jan/2020 11:05:34] "GET /favicon.ico HTTP/1.1" 404 77 [16/Jan/2020 11:05:39] "GET /admin/ HTTP/1.1" 302 0 [16/Jan/2020 11:05:39] "GET /admin/login/?next=/admin/ HTTP/1.1" 200 1870 [16/Jan/2020 11:05:39] "GET /static/admin/css/base.css HTTP/1.1" 404 77 [16/Jan/2020 11:05:39] "GET /static/admin/css/responsive.css HTTP/1.1" 404 77 [16/Jan/2020 11:05:39] "GET /static/admin/css/login.css HTTP/1.1" 404 77 Fatal Python error: Cannot recover from stack overflow.

Thread 0x00007f009bfff700 (most recent call first): File "/ecq/run/lib/python3.7/socket.py", line 589 in readinto File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/core/servers/basehttp.py", line 179 in handle_one_request File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/core/servers/basehttp.py", line 171 in handle File "/ecq/run/lib/python3.7/socketserver.py", line 720 in init File "/ecq/run/lib/python3.7/socketserver.py", line 360 in finish_request File "/ecq/run/lib/python3.7/socketserver.py", line 650 in process_request_thread File "/ecq/run/lib/python3.7/threading.py", line 870 in run File "/ecq/run/lib/python3.7/threading.py", line 926 in _bootstrap_inner File "/ecq/run/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x00007f00a8dee700 (most recent call first): File "/ecq/run/lib/python3.7/socket.py", line 589 in readinto File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/core/servers/basehttp.py", line 179 in handle_one_request File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/core/servers/basehttp.py", line 171 in handle File "/ecq/run/lib/python3.7/socketserver.py", line 720 in init File "/ecq/run/lib/python3.7/socketserver.py", line 360 in finish_request File "/ecq/run/lib/python3.7/socketserver.py", line 650 in process_request_thread File "/ecq/run/lib/python3.7/threading.py", line 870 in run File "/ecq/run/lib/python3.7/threading.py", line 926 in _bootstrap_inner File "/ecq/run/lib/python3.7/threading.py", line 890 in _bootstrap

Current thread 0x00007f00a99ed700 (most recent call first): File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/expressions.py", line 159 in getstate File "/ecq/run/lib/python3.7/copy.py", line 96 in copy File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/expressions.py", line 332 in copy File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/expressions.py", line 238 in resolve_expression File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/lookups.py", line 79 in process_lhs File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/lookups.py", line 153 in process_lhs File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/lookups.py", line 162 in as_sql File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/sql/compiler.py", line 405 in compile File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/sql/where.py", line 81 in as_sql File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/sql/compiler.py", line 405 in compile File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/sql/compiler.py", line 489 in as_sql File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/sql/compiler.py", line 1120 in execute_sql File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 55 in iter File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 1242 in _fetch_all File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 274 in iter File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 250 in repr File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/encoding.py", line 151 in transform File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/encoding.py", line 205 in shorten File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/base.py", line 174 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/init.py", line 68 in varmap File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/base.py", line 176 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/stacks.py", line 299 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/stacks.py", line 299 in get_frame_info File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/contrib/django/utils.py", line 84 in iterate_with_template_sources File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/contrib/django/client.py", line 207 in _get_stack_info_for_trace File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/base.py", line 169 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 432 in end File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 275 in end_span File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 599 in exit File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/instrumentation/packages/psycopg2.py", line 87 in call File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/instrumentation/packages/dbapi2.py", line 257 in call_if_sampling File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/postgresql/base.py", line 178 in get_new_connection File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 195 in connect File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 217 in ensure_connection File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 233 in _cursor File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 256 in cursor File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/sql/compiler.py", line 1131 in execute_sql File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 55 in iter File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 1242 in _fetch_all File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 274 in iter File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 250 in repr File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/encoding.py", line 151 in transform File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/encoding.py", line 205 in shorten File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/base.py", line 174 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/init.py", line 68 in varmap File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/base.py", line 176 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/stacks.py", line 299 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/stacks.py", line 299 in get_frame_info File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/contrib/django/utils.py", line 84 in iterate_with_template_sources File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/contrib/django/client.py", line 207 in _get_stack_info_for_trace File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/base.py", line 169 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 432 in end File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 275 in end_span File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 599 in exit File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/instrumentation/packages/psycopg2.py", line 87 in call File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/instrumentation/packages/dbapi2.py", line 257 in call_if_sampling File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/postgresql/base.py", line 178 in get_new_connection File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 195 in connect File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 217 in ensure_connection File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 233 in _cursor File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 256 in cursor File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/sql/compiler.py", line 1131 in execute_sql File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 55 in iter File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 1242 in _fetch_all File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 274 in iter File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 250 in repr File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/encoding.py", line 151 in transform File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/encoding.py", line 205 in shorten File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/base.py", line 174 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/init.py", line 68 in varmap File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/base.py", line 176 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/stacks.py", line 299 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/stacks.py", line 299 in get_frame_info File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/contrib/django/utils.py", line 84 in iterate_with_template_sources File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/contrib/django/client.py", line 207 in _get_stack_info_for_trace File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/base.py", line 169 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 432 in end File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 275 in end_span File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 599 in exit File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/instrumentation/packages/psycopg2.py", line 87 in call File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/instrumentation/packages/dbapi2.py", line 257 in call_if_sampling File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/postgresql/base.py", line 178 in get_new_connection File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 195 in connect File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 217 in ensure_connection File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 233 in _cursor File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 256 in cursor File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/sql/compiler.py", line 1131 in execute_sql File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 55 in iter File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 1242 in _fetch_all File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 274 in iter File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 250 in repr File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/encoding.py", line 151 in transform File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/encoding.py", line 205 in shorten File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/base.py", line 174 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/init.py", line 68 in varmap File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/base.py", line 176 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/stacks.py", line 299 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/stacks.py", line 299 in get_frame_info File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/contrib/django/utils.py", line 84 in iterate_with_template_sources File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/contrib/django/client.py", line 207 in _get_stack_info_for_trace ...

Thread 0x00007f00ab56c700 (most recent call first): File "/ecq/run/lib/python3.7/selectors.py", line 415 in select File "/ecq/run/lib/python3.7/socketserver.py", line 232 in serve_forever File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/core/servers/basehttp.py", line 213 in run File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/core/management/commands/runserver.py", line 139 in inner_run File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/utils/autoreload.py", line 54 in wrapper File "/ecq/run/lib/python3.7/threading.py", line 870 in run File "/ecq/run/lib/python3.7/threading.py", line 926 in _bootstrap_inner File "/ecq/run/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x00007f00b8a0d700 (most recent call first): File "/ecq/run/lib/python3.7/threading.py", line 300 in wait File "/ecq/run/lib/python3.7/threading.py", line 552 in wait File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/threading.py", line 74 in run File "/ecq/run/lib/python3.7/threading.py", line 926 in _bootstrap_inner File "/ecq/run/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x00007f00b920e700 (most recent call first): File "/ecq/run/lib/python3.7/threading.py", line 300 in wait File "/ecq/run/lib/python3.7/threading.py", line 552 in wait File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/threading.py", line 74 in run File "/ecq/run/lib/python3.7/threading.py", line 926 in _bootstrap_inner File "/ecq/run/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x00007f00b9a0f700 (most recent call first): File "/ecq/run/lib/python3.7/threading.py", line 300 in wait File "/ecq/run/lib/python3.7/queue.py", line 179 in get File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/transport/base.py", line 126 in _process_queue File "/ecq/run/lib/python3.7/threading.py", line 870 in run File "/ecq/run/lib/python3.7/threading.py", line 926 in _bootstrap_inner File "/ecq/run/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x00007f00ca798740 (most recent call first): File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/utils/autoreload.py", line 357 in tick File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/utils/autoreload.py", line 307 in run_loop File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/utils/autoreload.py", line 301 in run File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/utils/autoreload.py", line 583 in start_django File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/utils/autoreload.py", line 598 in run_with_reloader File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/core/management/commands/runserver.py", line 102 in run File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/core/management/commands/runserver.py", line 95 in handle File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/core/management/base.py", line 364 in execute File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/core/management/commands/runserver.py", line 60 in execute File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/core/management/base.py", line 323 in run_from_argv File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/core/management/init.py", line 375 in execute File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/core/management/init.py", line 381 in execute_from_command_line File "manage.py", line 17 in main File "manage.py", line 21 in … ———

Regarding Apache service side, here is the content of my mod_wsgi settings, once WSGIApplicationGroup set, no more as default, but according your advice: ——— … [/ecq/run/apache/conf/extra]# vi httpd-wsgi.conf

[/ecq/run/apache/conf/extra]# cat !$|grep -v "#" cat httpd-wsgi.conf|grep -v "#" WSGIApplicationGroup %{GLOBAL} WSGICaseSensitivity On WSGIPassAuthorization On WSGIPythonEggs /ecq/tmp/.python-eggs/ WSGIPythonHome /ecq/run/ WSGIPythonOptimize 1 WSGIPythonPath /ecq/run/ WSGISocketPrefix /ecq/run/apache/logs/ … ———

As result, once Apache restarted, as for the runserver, the Apache server is still sadly crashing when I try to log in Django admin GUI, with the following traces: ——— … [Thu Jan 16 11:07:59.358781 2020] [mpm_event:notice] [pid 1472:tid 140253765560128] AH00492: caught SIGWINCH, shutting down gracefully [Thu Jan 16 11:08:00.535760 2020] [mpm_event:notice] [pid 11457:tid 140385420093248] AH00489: Apache/2.4.41 (Unix) OpenSSL/1.1.1d mod_wsgi/4.7.0 Python/3.7 configured -- resuming normal operations [Thu Jan 16 11:08:00.535902 2020] [core:notice] [pid 11457:tid 140385420093248] AH00094: Command line: '/ecq/run/apache/bin/httpd' Fatal Python error: Cannot recover from stack overflow.

Current thread 0x00007fadf2ff5700 (most recent call first): File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/conf/init.py", line 388 in getattr File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/base.py", line 172 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/init.py", line 68 in varmap File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/base.py", line 176 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/stacks.py", line 299 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/stacks.py", line 299 in get_frame_info File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/contrib/django/utils.py", line 84 in iterate_with_template_sources File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/contrib/django/client.py", line 207 in _get_stack_info_for_trace File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/base.py", line 169 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 432 in end File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 275 in end_span File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 599 in exit File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/instrumentation/packages/psycopg2.py", line 87 in call File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/instrumentation/packages/dbapi2.py", line 257 in call_if_sampling File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/postgresql/base.py", line 178 in get_new_connection File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 195 in connect File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 217 in ensure_connection File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 233 in _cursor File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 256 in cursor File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/sql/compiler.py", line 1131 in execute_sql File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 55 in iter File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 1242 in _fetch_all File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 274 in iter File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 250 in repr File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/encoding.py", line 151 in transform File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/encoding.py", line 205 in shorten File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/base.py", line 174 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/init.py", line 68 in varmap File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/base.py", line 176 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/stacks.py", line 299 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/stacks.py", line 299 in get_frame_info File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/contrib/django/utils.py", line 84 in iterate_with_template_sources File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/contrib/django/client.py", line 207 in _get_stack_info_for_trace File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/base.py", line 169 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 432 in end File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 275 in end_span File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 599 in exit File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/instrumentation/packages/psycopg2.py", line 87 in call File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/instrumentation/packages/dbapi2.py", line 257 in call_if_sampling File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/postgresql/base.py", line 178 in get_new_connection File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 195 in connect File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 217 in ensure_connection File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 233 in _cursor File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 256 in cursor File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/sql/compiler.py", line 1131 in execute_sql File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 55 in iter File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 1242 in _fetch_all File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 274 in iter File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 250 in repr File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/encoding.py", line 151 in transform File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/encoding.py", line 205 in shorten File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/base.py", line 174 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/init.py", line 68 in varmap File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/base.py", line 176 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/stacks.py", line 299 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/stacks.py", line 299 in get_frame_info File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/contrib/django/utils.py", line 84 in iterate_with_template_sources File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/contrib/django/client.py", line 207 in _get_stack_info_for_trace File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/base.py", line 169 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 432 in end File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 275 in end_span File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 599 in exit File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/instrumentation/packages/psycopg2.py", line 87 in call File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/instrumentation/packages/dbapi2.py", line 257 in call_if_sampling File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/postgresql/base.py", line 178 in get_new_connection File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 195 in connect File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 217 in ensure_connection File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 233 in _cursor File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 256 in cursor File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/sql/compiler.py", line 1131 in execute_sql File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 55 in iter File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 1242 in _fetch_all File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 274 in iter File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 250 in repr File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/encoding.py", line 151 in transform File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/encoding.py", line 205 in shorten File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/base.py", line 174 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/init.py", line 68 in varmap File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/base.py", line 176 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/stacks.py", line 299 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/stacks.py", line 299 in get_frame_info File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/contrib/django/utils.py", line 84 in iterate_with_template_sources File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/contrib/django/client.py", line 207 in _get_stack_info_for_trace File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/base.py", line 169 in File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 432 in end File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 275 in end_span File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/traces.py", line 599 in exit File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/instrumentation/packages/psycopg2.py", line 87 in call File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/instrumentation/packages/dbapi2.py", line 257 in call_if_sampling File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/postgresql/base.py", line 178 in get_new_connection File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 195 in connect File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 217 in ensure_connection File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 233 in _cursor File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/backends/base/base.py", line 256 in cursor File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/sql/compiler.py", line 1131 in execute_sql File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 55 in iter File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 1242 in _fetch_all File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 274 in iter File "/ecq/run/lib/python3.7/site-packages/Django-2.2.9-py3.7.egg/django/db/models/query.py", line 250 in repr File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/encoding.py", line 151 in transform ...

Thread 0x00007fadf77fe700 (most recent call first):

Thread 0x00007fadf57fa700 (most recent call first):

Thread 0x00007fadf7fff700 (most recent call first):

Thread 0x00007faddaffd700 (most recent call first): File "/ecq/run/lib/python3.7/threading.py", line 300 in wait File "/ecq/run/lib/python3.7/threading.py", line 552 in wait File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/threading.py", line 74 in run File "/ecq/run/lib/python3.7/threading.py", line 926 in _bootstrap_inner File "/ecq/run/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x00007faddb7fe700 (most recent call first): File "/ecq/run/lib/python3.7/threading.py", line 300 in wait File "/ecq/run/lib/python3.7/threading.py", line 552 in wait File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/utils/threading.py", line 74 in run File "/ecq/run/lib/python3.7/threading.py", line 926 in _bootstrap_inner File "/ecq/run/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x00007faddbfff700 (most recent call first): File "/ecq/run/lib/python3.7/threading.py", line 300 in wait File "/ecq/run/lib/python3.7/queue.py", line 179 in get File "/ecq/run/lib/python3.7/site-packages/elastic_apm-5.3.3-py3.7-linux-x86_64.egg/elasticapm/transport/base.py", line 126 in _process_queue File "/ecq/run/lib/python3.7/threading.py", line 870 in run File "/ecq/run/lib/python3.7/threading.py", line 926 in _bootstrap_inner File "/ecq/run/lib/python3.7/threading.py", line 890 in _bootstrap

Thread 0x00007fadfcc6c700 (most recent call first):

Thread 0x00007fae0718a740 (most recent call first): [Thu Jan 16 11:08:26.606478 2020] [core:notice] [pid 11457:tid 140385420093248] AH00052: child pid 11463 exit signal Aborted (6)

… ———

I hope this match to your exactions and may help you… hum… to help me ! ;-)

Thank you very much for your help and feedback and have a good day.

All the best,

Vincent

beniwohli commented 4 years ago

Thanks @vclaudic, this helps a lot. I think I'm starting to see what is happening here. Looking at the stack traces, it seems like there is an infinite recursion going on (notice how the frames start to repeat after a while). Normally, Python should raise a RecursionError at some point, but apparently doesn't always do that, depending on circumstances I haven't quite understood yet (see e.g. https://bugs.python.org/issue37402), and instead crashes.

Now, the next question is, why is there a recursion error. Based on the stack frames, it looks like we're collecting local variables for a database span, and one of the local variables is a queryset. To show a representation of the queryset, we call repr() on it, which causes a database query, which we again capture in a span, which collects local variables, one of which is a queryset.... and we're in an infinite recursion loop. This is bad, and we need to fix it.

But first, a possible quick workaround. Collection of local variables can be configured using the collect_local_variables setting. By default, we shouldn't collect local variables for transactions/spans. Did you maybe change that setting to transactions or all? If yes, I suggest trying it with errors. Otherwise, try to disable it completely by setting it to off.

Now, to actually fix the issue, we need to special-case QuerySets in our transform function, similarly how we already special-case some other types.

vclaudic commented 4 years ago

Hi Benjamin,

Clap ! clap ! I'm really impressed !

I've tested your workaround and I may say that from both points of view, i.e. Apache and runserver, it works fine ! I've changed my setting according your advice and replace "all" with "errors" and yet the admin login doesn't crash both Apache and runserver.

I really thank you verrrrrrrrrry much for your support, feedback and reactivity ! ;-)

Have a good day.

All the best,

Vincent

beniwohli commented 4 years ago

@vclaudic that's great news, thanks for reporting back! We already have an issue to improve upon the transform function (https://github.com/elastic/apm-agent-python/issues/182), so I'll go ahead and close this one :)