search

unbit / uwsgi

uWSGI application server container
http://projects.unbit.it/uwsgi
Other
3.45k stars 686 forks source link

uwsgi hangs forever when restarting or reloading worker using C++ extension #2023

Open draganHR opened 5 years ago

draganHR commented 5 years ago

I'm having a problem with restarting and reloading uwsgi when python2.7 application imports certain C++ extension. If i don't import the extension module, everything works fine, but if module is loaded, uwsgi restart hangs forever.

$ uwsgi --module example:application --socket /tmp/uwsgi.sock --processes=4 --master --worker-reload-mercy=15
*** Starting uWSGI 2.0.18 (64bit) on [Thu May 23 09:26:11 2019] ***
compiled with version: 4.8.5 20150623 (Red Hat 4.8.5-36) on 23 May 2019 07:09:57
os: Linux-3.10.0-957.10.1.el7.x86_64 #1 SMP Mon Mar 18 15:06:45 UTC 2019
nodename: tcbase-vagrant
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 2
current working directory: /vagrant/uwsgi-demo
detected binary path: /vagrant/uwsgi-demo/.env/bin/uwsgi
your processes number limit is 4096
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to UNIX address /tmp/uwsgi.sock fd 3
Python version: 2.7.5 (default, Apr  9 2019, 14:30:50)  [GCC 4.8.5 20150623 (Red Hat 4.8.5-36)]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x19d7e00
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 15 seconds
mapped 364600 bytes (356 KB) for 4 cores
*** Operational MODE: preforking ***
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x19d7e00 pid: 21734 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 21734)
spawned uWSGI worker 1 (pid: 21736, cores: 1)
spawned uWSGI worker 2 (pid: 21737, cores: 1)
spawned uWSGI worker 3 (pid: 21738, cores: 1)
spawned uWSGI worker 4 (pid: 21739, cores: 1)
^CSIGINT/SIGQUIT received...killing workers...
Thu May 23 09:26:53 2019 - worker 1 (pid: 21736) is taking too much time to die...NO MERCY !!!
Thu May 23 09:26:53 2019 - worker 2 (pid: 21737) is taking too much time to die...NO MERCY !!!
Thu May 23 09:26:53 2019 - worker 3 (pid: 21738) is taking too much time to die...NO MERCY !!!
Thu May 23 09:26:53 2019 - worker 4 (pid: 21739) is taking too much time to die...NO MERCY !!!
worker 1 buried after 1 seconds
worker 2 buried after 1 seconds
worker 3 buried after 1 seconds
worker 4 buried after 1 seconds
goodbye to uWSGI.

Unfortunately the extension is propriatary so i cannot provide it.

Interesting thing is that extension starts a thread as soon the module is imported. The thread is started only in master proces, except if i use --lazy, but that does not solve the issue.

$ ps -o command,thcount | grep uwsgi
uwsgi --module example     2
uwsgi --module example     1
uwsgi --module example     1
uwsgi --module example     1
uwsgi --module example     1

Stats server reports all workers as idle. I tried uwsgi versions 2.0.18 from pypi and 2.0.17.1-2.el7 (EPEL)

Any idea why uwsgi won't restart the workers?

SB2020-eye commented 4 years ago

Same issue here