search

unbit / uwsgi

uWSGI application server container
http://projects.unbit.it/uwsgi
Other
3.45k stars 691 forks source link

Segmentation fault in python 3.7, uwsgi 2.0.19.1 #2281

Closed irisu-inwl closed 3 years ago

irisu-inwl commented 3 years ago

Segmentation fault occurs in the API deployed on GKE. I don't know why it happens, so I would like to know the cause and how to fix it.

$ pip freeze
alembic==1.0.5
BigQuery-Python==1.15.0
cachetools==4.2.0
certifi==2020.12.5
cffi==1.14.4
chardet==4.0.0
Click==7.0
Flask==1.0.2
Flask-Migrate==2.3.1
Flask-SQLAlchemy==2.3.2
gcp-accessor==0.0.6
google-api-core==1.25.0
google-api-python-client==1.12.8
google-auth==1.24.0
google-auth-httplib2==0.0.4
google-cloud-bigquery==2.6.2
google-cloud-core==1.5.0
google-cloud-firestore==2.0.2
google-cloud-pubsub==2.2.0
google-cloud-storage==1.35.0
google-crc32c==1.1.2
google-resumable-media==1.2.0
googleapis-common-protos==1.52.0
grpc-google-iam-v1==0.12.3
grpcio==1.35.0
httplib2==0.18.1
idna==2.10
isodate==0.6.0
itsdangerous==1.1.0
Jinja2==2.10
jsonschema==2.6.0
libcst==0.3.16
Mako==1.0.7
MarkupSafe==1.1.0
mypy-extensions==0.4.3
mysqlclient==1.3.13
numpy==1.19.5
oauth2client==3.0.0
pandas==0.23.4
proto-plus==1.13.0
protobuf==3.14.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.20
python-dateutil==2.7.5
python-editor==1.0.3
pytz==2018.7
PyYAML==5.4.1
requests==2.25.1
rsa==4.7
scikit-learn==0.20.3
scipy==1.6.0
six==1.15.0
SQLAlchemy==1.2.14
typing-extensions==3.7.4.3
typing-inspect==0.6.0
uritemplate==3.0.1
urllib3==1.26.2
uWSGI==2.0.19.1
Werkzeug==0.15.3
xlrd==1.2.0
[uwsgi]
chdir = /code/python
wsgi-file = wsgi.py
callable = app
master = true
processes = 2
threads = 4
socket = :3031
uid=1000
gid=1000
chmod-socket = 666
pidfile = /code/python/uwsgi.pid
vacuum = true
die-on-term = true
uwsgi --emperor python/uwsgi.ini

!!! uWSGI process 210 got Segmentation Fault !!!
*** backtrace of 210 ***
/usr/local/bin/uwsgi(uwsgi_backtrace+0x2a) [0x55aaa614605a]
/usr/local/bin/uwsgi(uwsgi_segfault+0x23) [0x55aaa6146413]
/lib/x86_64-linux-gnu/libc.so.6(+0x37840) [0x7f8057ea9840]
/usr/local/lib/python3.7/site-packages/grpc/_cython/cygrpc.cpython-37m-x86_64-linux-gnu.so(+0x2b19d5) [0x7f80530709d5]
/usr/local/lib/python3.7/site-packages/grpc/_cython/cygrpc.cpython-37m-x86_64-linux-gnu.so(+0x2a8a72) [0x7f8053067a72]
/usr/local/lib/python3.7/site-packages/grpc/_cython/cygrpc.cpython-37m-x86_64-linux-gnu.so(+0x2a98e9) [0x7f80530688e9]
/usr/local/lib/python3.7/site-packages/grpc/_cython/cygrpc.cpython-37m-x86_64-linux-gnu.so(+0x18e4fd) [0x7f8052f4d4fd]
/usr/local/lib/python3.7/site-packages/grpc/_cython/cygrpc.cpython-37m-x86_64-linux-gnu.so(+0x2ae244) [0x7f805306d244]
/usr/local/lib/python3.7/site-packages/grpc/_cython/cygrpc.cpython-37m-x86_64-linux-gnu.so(+0x2c5009) [0x7f8053084009]
/usr/local/lib/python3.7/site-packages/grpc/_cython/cygrpc.cpython-37m-x86_64-linux-gnu.so(+0x29df28) [0x7f805305cf28]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3) [0x7f8058d25fa3]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f8057f6b4cf]
*** end of backtrace ***
Tue Jan 26 02:12:56 2021 - uWSGI worker 2 screams: UAAAAAAH my master disconnected: i will kill myself !!!
Tue Jan 26 02:12:56 2021 - uWSGI worker 1 screams: UAAAAAAH my master disconnected: i will kill myself !!!
Tue Jan 26 02:12:56 2021 - uWSGI worker 2 screams: UAAAAAAH my master disconnected: i will kill myself !!!
Tue Jan 26 02:12:56 2021 - uWSGI worker 1 screams: UAAAAAAH my master disconnected: i will kill myself !!!
Tue Jan 26 02:12:56 2021 - uWSGI worker 2 screams: UAAAAAAH my master disconnected: i will kill myself !!!
Tue Jan 26 02:12:56 2021 - uWSGI worker 1 screams: UAAAAAAH my master disconnected: i will kill myself !!!
Tue Jan 26 02:12:56 2021 - uWSGI worker 2 screams: UAAAAAAH my master disconnected: i will kill myself !!!
Tue Jan 26 02:12:56 2021 - uWSGI worker 1 screams: UAAAAAAH my master disconnected: i will kill myself !!!
Tue Jan 26 02:12:56 2021 - [emperor] curse the uwsgi instance /code/python/uwsgi.ini (pid: 210)
Tue Jan 26 02:12:59 2021 - [emperor] removed uwsgi instance /code/python/uwsgi.ini
*** has_emperor mode detected (fd: 5) ***
xrmx commented 3 years ago

There's something wrong going on with the grpc module. Give it a try with threads = 1

irisu-inwl commented 3 years ago

@xrmx Thanks, but I got the same error with threads = 1. 

!!! uWSGI process 59 got Segmentation Fault !!!
*** backtrace of 59 ***
/usr/local/bin/uwsgi(uwsgi_backtrace+0x2a) [0x5579aca7505a]
/usr/local/bin/uwsgi(uwsgi_segfault+0x23) [0x5579aca75413]
/lib/x86_64-linux-gnu/libc.so.6(+0x37840) [0x7f3330d35840]
/usr/local/lib/python3.7/site-packages/grpc/_cython/cygrpc.cpython-37m-x86_64-linux-gnu.so(+0x2b19d5) [0x7f332bf119d5]
/usr/local/lib/python3.7/site-packages/grpc/_cython/cygrpc.cpython-37m-x86_64-linux-gnu.so(+0x2a8a72) [0x7f332bf08a72]
/usr/local/lib/python3.7/site-packages/grpc/_cython/cygrpc.cpython-37m-x86_64-linux-gnu.so(+0x2a98e9) [0x7f332bf098e9]
/usr/local/lib/python3.7/site-packages/grpc/_cython/cygrpc.cpython-37m-x86_64-linux-gnu.so(+0x18e4fd) [0x7f332bdee4fd]
/usr/local/lib/python3.7/site-packages/grpc/_cython/cygrpc.cpython-37m-x86_64-linux-gnu.so(+0x2ae244) [0x7f332bf0e244]
/usr/local/lib/python3.7/site-packages/grpc/_cython/cygrpc.cpython-37m-x86_64-linux-gnu.so(+0x2c5009) [0x7f332bf25009]
/usr/local/lib/python3.7/site-packages/grpc/_cython/cygrpc.cpython-37m-x86_64-linux-gnu.so(+0x29df28) [0x7f332befdf28]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3) [0x7f3331bb1fa3]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f3330df74cf]
*** end of backtrace ***
Tue Jan 26 08:24:06 2021 - uWSGI worker 2 screams: UAAAAAAH my master disconnected: i will kill myself !!!
Tue Jan 26 08:24:06 2021 - uWSGI worker 1 screams: UAAAAAAH my master disconnected: i will kill myself !!!
Tue Jan 26 08:24:07 2021 - [emperor] curse the uwsgi instance /code/python/uwsgi.ini (pid: 59)
Tue Jan 26 08:24:10 2021 - [emperor] removed uwsgi instance /code/python/uwsgi.ini
*** has_emperor mode detected (fd: 5) ***
[uWSGI] getting INI configuration from /code/python/uwsgi.ini

I'm going to try to update the firestore library, since most of the time when the error occurs, the API calls firestore.batch.commit().

xrmx commented 3 years ago

The give a try to lazy-apps, grpc is probably doing things with threads before workers are forked.

irisu-inwl commented 3 years ago

I set lazy-apps = true and after 150 requests, no request errors! Thank you very much! I'll close the issue!