search

unbit / uwsgi

uWSGI application server container
http://projects.unbit.it/uwsgi
Other
3.45k stars 691 forks source link

SEGVs + WSGIRequest object corruption on master #1478

Open unixwitch opened 7 years ago

unixwitch commented 7 years ago

Using master@f262899bc5eb890007ba157d8b1da1e836049b74, we've been seeing strange problems with Django applications. The first is that WSGIRequest will be randomly missing attributes it should have:

AttributeError: 'WSGIRequest' object has no attribute 'environ'

or

AttributeError: 'WSGIRequest' object has no attribute 'path'

The second is intermittent SEGVs:

[kidneycareukwagtail] - !!! uWSGI process 16106 got Segmentation Fault !!!
[kidneycareukwagtail] - *** backtrace of 16106 ***
[kidneycareukwagtail] uWSGI worker 1(uwsgi_backtrace+0x30) [0x46a040]
[kidneycareukwagtail] uWSGI worker 1(uwsgi_segfault+0x21) [0x46a441]
/lib/x86_64-linux-gnu/libc.so.6(+0x350e0) [0x7f36294e70e0]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(+0x19295d) [0x7f362890c95d]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(+0xa67af) [0x7f36288207af]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(+0x192c27) [0x7f362890cc27]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(_PyObject_GC_Malloc+0xeb) [0x7f362890de9b]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyType_GenericAlloc+0x3b) [0x7f362883ef0b]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(+0xca223) [0x7f3628844223]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyObject_Call+0x68) [0x7f36287e1b48]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyEval_EvalFrameEx+0x4bcc) [0x7f36288c33fc]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyEval_EvalCodeEx+0x886) [0x7f36288c70f6]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyEval_EvalFrameEx+0x608a) [0x7f36288c48ba]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyEval_EvalCodeEx+0x886) [0x7f36288c70f6]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(+0x94c4c) [0x7f362880ec4c]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyObject_Call+0x68) [0x7f36287e1b48]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(+0x7ebed) [0x7f36287f8bed]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyObject_Call+0x68) [0x7f36287e1b48]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(+0xcc7e9) [0x7f36288467e9]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(+0xca266) [0x7f3628844266]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyObject_Call+0x68) [0x7f36287e1b48]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyEval_EvalFrameEx+0x4bcc) [0x7f36288c33fc]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyEval_EvalCodeEx+0x886) [0x7f36288c70f6]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(+0x94c4c) [0x7f362880ec4c]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyObject_Call+0x68) [0x7f36287e1b48]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(+0x7ebed) [0x7f36287f8bed]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyObject_Call+0x68) [0x7f36287e1b48]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(+0xcc7e9) [0x7f36288467e9]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(+0xca266) [0x7f3628844266]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyObject_Call+0x68) [0x7f36287e1b48]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyEval_EvalFrameEx+0x4bcc) [0x7f36288c33fc]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyEval_EvalFrameEx+0x70a8) [0x7f36288c58d8]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyEval_EvalFrameEx+0x70a8) [0x7f36288c58d8]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyEval_EvalCodeEx+0x886) [0x7f36288c70f6]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyEval_EvalFrameEx+0x608a) [0x7f36288c48ba]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyEval_EvalCodeEx+0x886) [0x7f36288c70f6]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyEval_EvalFrameEx+0x608a) [0x7f36288c48ba]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyEval_EvalFrameEx+0x70a8) [0x7f36288c58d8]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyEval_EvalCodeEx+0x886) [0x7f36288c70f6]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyEval_EvalFrameEx+0x608a) [0x7f36288c48ba]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyEval_EvalFrameEx+0x70a8) [0x7f36288c58d8]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyEval_EvalCodeEx+0x886) [0x7f36288c70f6]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(+0x94c4c) [0x7f362880ec4c]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyObject_Call+0x68) [0x7f36287e1b48]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(+0x7ebed) [0x7f36287f8bed]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyObject_Call+0x68) [0x7f36287e1b48]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(+0xcc7e9) [0x7f36288467e9]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(+0xca266) [0x7f3628844266]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyObject_Call+0x68) [0x7f36287e1b48]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyEval_EvalFrameEx+0x4bcc) [0x7f36288c33fc]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyEval_EvalCodeEx+0x886) [0x7f36288c70f6]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyEval_EvalFrameEx+0x608a) [0x7f36288c48ba]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyEval_EvalCodeEx+0x886) [0x7f36288c70f6]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyEval_EvalFrameEx+0x608a) [0x7f36288c48ba]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyEval_EvalFrameEx+0x70a8) [0x7f36288c58d8]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyEval_EvalFrameEx+0x70a8) [0x7f36288c58d8]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyEval_EvalFrameEx+0x70a8) [0x7f36288c58d8]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyEval_EvalFrameEx+0x70a8) [0x7f36288c58d8]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyEval_EvalFrameEx+0x70a8) [0x7f36288c58d8]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyEval_EvalFrameEx+0x70a8) [0x7f36288c58d8]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyEval_EvalFrameEx+0x70a8) [0x7f36288c58d8]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyEval_EvalCodeEx+0x886) [0x7f36288c70f6]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyEval_EvalFrameEx+0x608a) [0x7f36288c48ba]
/opt/tbx/python/3.4/lib/libpython3.4m.so.1.0(PyEval_EvalFrameEx+0x70a8) [0x7f36288c58d8]
*** end of backtrace ***

And other evidence of memory corruption:

*** Error in `[kidneycareukwagtail] uWSGI worker 1': double free or corruption (!prev): 0x00000000021b9430 ***
[kidneycareukwagtail] - DAMN ! worker 1 (pid: 15927) died, killed by signal 6 :( trying respawn ...

We've only seen this on our staging sites, which use master. Production sites using a 2.0-based branch don't seem to be affected. At least two affected applications use Python 3.4, but we've seen 'WSGIRequest' object has no attribute 'path' on 2.7 as well, so I don't think it's related to version. (We'll try updating one application to a newer Python version anyway.)

These problems are all intermittent and don't seem to have an obvious cause; they manifest as internal server errors to users, but the application works fine after the worker is restarted. I'm not entirely sure this is caused by uWSGI (it could be some Python or C module bug...), but the fact it only appears in 2.1 suggests it may be.

Here's an example uwsgi config of an affected application:

# vim:sw=4 ts=4 et:

[uwsgi]
binary-path         = /opt/tbx/bin/uwsgi
plugin-dir          = /opt/tbx/lib/uwsgi/plugins
plugin              = router_fcgi
plugin              = forkptyrouter

strict              = true
auto-procname       = true
procname-prefix-spaced = [kidneycareukwagtail]
protocol            = uwsgi
legion              = kidneycareukwagtail 239.64.245.218:45132 98 bf-cbc:redacted
legion-node         = kidneycareukwagtail 239.64.245.218:45132
memory-report       = true
die-on-term         = true
umask               = 022
no-orphans          = true
honour-range        = true

stats               = /var/www/kidneycareukwagtail/tmp/uwsgi-stats.sock

plugin              = python34

virtualenv          = /var/www/kidneycareukwagtail/venv

socket = /var/www/kidneycareukwagtail/tmp/uwsgi.socket
chmod-socket = 777

websockets-pong-tolerance = 30

master              = true
chdir               = /var/www/kidneycareukwagtail/app

check-static        = /var/www/kidneycareukwagtail/shared/public
check-static        = /var/www/kidneycareukwagtail/app/public

static-index        = index.html
static-index        = index.htm

static-map          = /media/images/=/var/www/kidneycareukwagtail/media/images/

route-label         = begin

# HTTP authentication
route-label         = auth

# IP whitelist

route-label         = auth-done

enable-threads      = true
offload-threads     = 2

cheaper-algo        = busyness
cheaper-overload    = 30
cheaper             = 1
cheaper-initial     = 1
cheaper-step        = 1

idle                = 3600

workers             = 9

env                 = PATH=/var/www/kidneycareukwagtail/venv/bin:/opt/tbx/python/3.4/bin:/opt/tbx/postgres/9.4/bin:/opt/tbx/sbin:/opt/tbx/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin
env                 = XDG_RUNTIME_DIR=/run/user/576142912
env                 = D=/var/www/kidneycareukwagtail/app
env                 = CFG_APPNAME=kidneycareukwagtail
env                 = CFG_APP_NAME=kidneycareukwagtail
env                 = CFG_HOSTNAMES=kidneycareuk-staging.torchboxapps.com
env                 = CFG_SECRET_KEY=redacted
env                 = CFG_SERVER_EMAIL=kidneycareukwagtail@staging-1-a.bmyrk.torchbox.net
env                 = CFG_EMAIL_SUBJECT_PREFIX=[kidneycareukwagtail@bmyrk/staging-1-a]
env                 = CFG_ENVIRONMENT_TYPE=staging
env                 = CFG_PUBLIC_DIR=/var/www/kidneycareukwagtail/shared/public
env                 = CFG_SHARED_DIR=/var/www/kidneycareukwagtail/shared
env                 = CFG_LOG_DIR=/var/www/kidneycareukwagtail/log
env                 = TMPDIR=/var/www/kidneycareukwagtail/tmp
env                 = TMP=/var/www/kidneycareukwagtail/tmp
env                 = TEMP=/var/www/kidneycareukwagtail/tmp
env                 = TEMPDIR=/var/www/kidneycareukwagtail/tmp
env                 = CFG_LOGFILE=/var/www/kidneycareukwagtail/log/error.log
env                 = CFG_ERROR_LOG=/var/www/kidneycareukwagtail/log/error.log
env                 = CFG_SERVER_NAME=kidneycareuk-staging.torchboxapps.com
env                 = CFG_SERVERALIASES=
env                 = CFG_ALLOWED_HOSTS=kidneycareuk-staging.torchboxapps.com
env                 = CFG_EMAIL_PREFIX=[kidneycareukwagtail@bmyrk/staging-1-a]
env                 = V=/var/www/kidneycareukwagtail/venv
env                 = PYTHONPATH=/var/www/kidneycareukwagtail/app
env                 = DATABASE_URL=postgres://%%2Fvar%%2Fwww%%2Fkidneycareukwagtail%%2Ftmp/kidneycareukwagtail
env                 = PGUSER=kidneycareukwagtail
env                 = PGDATABASE=kidneycareukwagtail
env                 = PGHOST=/var/www/kidneycareukwagtail/tmp
env                 = CFG_DB_USER=kidneycareukwagtail
env                 = CFG_DB_NAME=kidneycareukwagtail
env                 = CFG_DB_HOST=/var/www/kidneycareukwagtail/tmp
env                 = CFG_DB_TYPE=postgres
env                 = CFG_MEDIA_DIR=/var/www/kidneycareukwagtail/media/
env                 = CFG_MEDIA_URL=/media/
env                 = CFG_ELASTICSEARCH_URL=http://localhost:9200/
env                 = CACHE_PURGE_URL=http://193.227.244.102:80
env                 = DJANGO_SETTINGS_MODULE=kidneycareuk.settings.production

module          = kidneycareuk.wsgi:application

logto               = /var/www/kidneycareukwagtail/log/uwsgi.log
log-date            = true
log-prefix          = [kidneycareukwagtail]
log-maxsize         = 10485760
logformat           = %(addr) %(host) [%(ltime)] "%(method) %(uri) %(proto)" %(status) %(size) %(msecs)ms pid=%(pid) worker=%(wid) core=%(core) switches=%(switches) rss=%(rssM)M vsz=%(vszM)M
close-on-exec       = true
buffer-size         = 16384
thunder-lock        = true
pcre-jit            = true
uid                 = kidneycareukwagtail
gid                 = kidneycareukwagtail

I'm not really sure where to start with debugging this - at the point the crash happens, the problem has presumably already occurred, and I can't reproduce it...

cameronmaske commented 7 years ago

@unixwitch We are seeing similar issues with our Django application. We'll see random AttributeError in really unexpected parts of the code, like from Django Rest Framework....

AttributeError: rest_framework.fields in get_attribute

But also seen the same error....

AttributeError: 'WSGIRequest' object has no attribute 'path'

Followed by a very similar core dump.

Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: !!! uWSGI process 95 got Segmentation Fault !!!
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: *** backtrace of 95 ***
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: uwsgi(uwsgi_backtrace+0x30) [0x468300]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: uwsgi(uwsgi_segfault+0x21) [0x4686a1]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /lib/x86_64-linux-gnu/libc.so.6(+0x35180) [0x7fd97ee61180]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(+0x97a94) [0x7fd97f4a3a94]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(_PyObject_GenericSetAttrWithDict+0x132) [0x7fd97f4ab9e2]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyObject_SetAttr+0x8f) [0x7fd97f4ab36f]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x24d1) [0x7fd97f511db1]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x830) [0x7fd97f516730]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(+0x845cc) [0x7fd97f4905cc]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fd97f45ecb3]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyObject_CallFunction+0xbb) [0x7fd97f45ee0b]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(+0x7267a) [0x7fd97f47e67a]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(_PyObject_GenericSetAttrWithDict+0x107) [0x7fd97f4ab9b7]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyObject_SetAttr+0x8f) [0x7fd97f4ab36f]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x24d1) [0x7fd97f511db1]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x830) [0x7fd97f516730]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x594e) [0x7fd97f51522e]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x830) [0x7fd97f516730]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(+0x845cc) [0x7fd97f4905cc]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fd97f45ecb3]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyObject_CallFunction+0xbb) [0x7fd97f45ee0b]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(_PyObject_GenericGetAttrWithDict+0xb5) [0x7fd97f4ab655]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fd97f45ecb3]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x3b3e) [0x7fd97f51341e]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x830) [0x7fd97f516730]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(+0x845cc) [0x7fd97f4905cc]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fd97f45ecb3]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(+0x616cd) [0x7fd97f46d6cd]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fd97f45ecb3]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(+0xbf0c8) [0x7fd97f4cb0c8]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x163d) [0x7fd97f510f1d]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x830) [0x7fd97f516730]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x594e) [0x7fd97f51522e]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x830) [0x7fd97f516730]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(+0x84695) [0x7fd97f490695]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fd97f45ecb3]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x11c3) [0x7fd97f510aa3]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x830) [0x7fd97f516730]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(+0x84695) [0x7fd97f490695]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fd97f45ecb3]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x11c3) [0x7fd97f510aa3]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x830) [0x7fd97f516730]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(+0x84695) [0x7fd97f490695]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fd97f45ecb3]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x11c3) [0x7fd97f510aa3]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x830) [0x7fd97f516730]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(+0x845cc) [0x7fd97f4905cc]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fd97f45ecb3]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyObject_CallFunctionObjArgs+0x137) [0x7fd97f45f797]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/python2.7/site-packages/newrelic-2.70.0.51/newrelic/packages/wrapt/_wrappers.so(+0x42bd) [0x7fd97a5882bd]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fd97f45ecb3]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x11c3) [0x7fd97f510aa3]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x830) [0x7fd97f516730]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(+0x84695) [0x7fd97f490695]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fd97f45ecb3]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x11c3) [0x7fd97f510aa3]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x830) [0x7fd97f516730]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(+0x84695) [0x7fd97f490695]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fd97f45ecb3]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalFrameEx+0x11c3) [0x7fd97f510aa3]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyEval_EvalCodeEx+0x830) [0x7fd97f516730]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(+0x845cc) [0x7fd97f4905cc]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyObject_Call+0x43) [0x7fd97f45ecb3]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: /usr/local/lib/libpython2.7.so.1.0(PyObject_CallFunctionObjArgs+0x137) [0x7fd97f45f797]
Mar 15 14:47:49 ip-172-31-19-222.ec2.internal docker[31241]: *** end of backtrace ***

Packages used: Python==2.7.10 Django==1.8.10 uWSGI==2.0.12

Are you by chance using the newrelic python wrapper? We've had a bunch of AttributeError surface through that. Which version of Python 2.7 did you see this happen on?

Our uswgi config looks like...

[uwsgi]
master = true
processes = 4
http  = :80
module = app.config.wsgi
enable-threads = true
single-interpreter = true
wsgi-env-behavior = holy
die-on-term = true
harakiri = 60
harakiri-verbose = true
buffer-size = 65535
chdir = /code/
unbit commented 7 years ago

So this happens even on 2.0, but maybe is related to the 'holy' env behaviour ? (in 2.1 became the default setting). I will investigate in this area

unixwitch commented 7 years ago

We don't use New Relic, but we do use Raven (the Sentry client) which does some things to the WSGI request. I've asked a developer to see if the problem persists after removing Raven, but I didn't hear back yet.

unixwitch commented 7 years ago

Ran into a similar-looking problem with another Django application, Python 3.6:

AttributeError: 'SessionStore' object has no attribute '_SessionBase__session_key'
Internal Server Error: /
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 235, in get_response
    response = middleware_method(request, response)
  File "/usr/local/lib/python3.6/site-packages/django/contrib/messages/middleware.py", line 23, in process_response
    unstored_messages = request._messages.update(response)
  File "/usr/local/lib/python3.6/site-packages/django/contrib/messages/storage/base.py", line 137, in update
    self._prepare_messages(self._queued_messages)
AttributeError: 'FallbackStorage' object has no attribute '_queued_messages'
[pid: 15|app: 0|req: 7/14] 172.31.240.26 () {46 vars in 1669 bytes} [Thu Mar 23 11:17:33 2017] GET / => generated 27 bytes in 40 msecs (HTTP/1.1 500) 1 headers in 63 bytes (1 switches on core 0)
Not Found: /favicon.ico
[pid: 12|app: 0|req: 8/15] 172.31.240.26 () {42 vars in 1073 bytes} [Thu Mar 23 11:17:33 2017] GET /favicon.ico => generated 85 bytes in 44 msecs (HTTP/1.1 404) 2 headers in 80 bytes (1 switches on core 0)
Thu Mar 23 11:20:24 2017 - !!! uWSGI process 15 got Segmentation Fault !!!
Thu Mar 23 11:20:24 2017 - DAMN ! worker 2 (pid: 15) died :( trying respawn ...

I've switched it to use --wsgi-env-behaviour=cheat to see whether that makes any difference.

unixwitch commented 7 years ago

The users report that setting wsgi-env-behaviour = cheat on our 2.1-based sites has fixed the problem.

Removing Raven didn't have any effect, so I don't think that's related.

paultag commented 7 years ago

I was having the same issue (lots of SIGSEGV), and I can confirm @unixwitch's hack (--wsgi-env-behaviour cheat) worked