Something odd happens if you try to join a thread in an atexit handler.

Consider the following code:

import atexit
import threading

def make_app():
    def application(env, start_response):
        start_response('200 OK', [('Content-Type', 'text/html')])
        return [b"Hello World"]

    e = threading.Event()

    def bg_thread():
        print("thread started")
        e.wait()
        print("thread exiting")

    t = threading.Thread(target=bg_thread, daemon=True)
    t.start()

    @atexit.register
    def cleanup():
        print("start clean up")
        e.set()
        print("msg sent")
        t.join()
        print("end clean up")

    return application

All it does is starting a thread in the background that waits for an event to be set.

If you run it with uwsgi --enable-threads --http-socket=127.0.0.1:8000 --module "mywsgi:make_app()" everything works as espected:

$ uwsgi --enable-threads --http-socket=127.0.0.1:8000 --module "mywsgi:make_app()"
*** Starting uWSGI 2.0.15 (64bit) on [Mon Aug 14 13:50:52 2017] ***
compiled with version: 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42) on 21 July 2017 12:40:07
os: Darwin-16.7.0 Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64
nodename: alpha.local
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 8
current working directory: /Users/hynek/Projects/ssce
detected binary path: /Users/hynek/.virtualenvs/ssce/bin/uwsgi
*** WARNING: you are running uWSGI without its master process manager ***
your processes number limit is 709
your memory page size is 4096 bytes
detected max file descriptor number: 7168
lock engine: OSX spinlocks
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 127.0.0.1:8000 fd 3
Python version: 3.6.1 (default, May  4 2017, 15:25:00)  [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)]
Python main interpreter initialized at 0x7fda0c000000
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 72752 bytes (71 KB) for 1 cores
*** Operational MODE: single process ***
thread started
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x7fda0c000000 pid: 93411 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI worker 1 (and the only) (pid: 93411, cores: 1)
^Cstart clean up
msg sent
thread exiting
end clean up

However if you run it with a master process, it hangs while waiting for the thread which never does receive it:

$ uwsgi --master --enable-threads --http-socket=127.0.0.1:8000 --module "mywsgi:make_app()"
*** Starting uWSGI 2.0.15 (64bit) on [Mon Aug 14 13:53:08 2017] ***
compiled with version: 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42) on 21 July 2017 12:40:07
os: Darwin-16.7.0 Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64
nodename: alpha.local
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 8
current working directory: /Users/hynek/Projects/ssce
detected binary path: /Users/hynek/.virtualenvs/ssce/bin/uwsgi
your processes number limit is 709
your memory page size is 4096 bytes
detected max file descriptor number: 7168
lock engine: OSX spinlocks
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 127.0.0.1:8000 fd 3
Python version: 3.6.1 (default, May  4 2017, 15:25:00)  [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)]
Python main interpreter initialized at 0x7ff63702a000
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 145504 bytes (142 KB) for 1 cores
*** Operational MODE: single process ***
thread started
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x7ff63702a000 pid: 93425 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 93425)
spawned uWSGI worker 1 (pid: 93426, cores: 1)
^CSIGINT/SIGQUIT received...killing workers...
start clean up
msg sent
Mon Aug 14 13:54:12 2017 - worker 1 (pid: 93426) is taking too much time to die...NO MERCY !!!
worker 1 buried after 1 seconds
goodbye to uWSGI.

I guess threading is disabled too soon or something? This is relevant to me because I need to clean up my background thread in prometheus_async.

Hi, this is the expected behaviour with threads + fork (and it is one of the main reasons why golang does not support it ;) Basically the thread is spawned in the master, but the wait happens in the worker for a thread that is not 'runnable'. --lazy-apps should fix your problem. If you need copy on write, spawn the thread in a post_fork hook

It does, thanks for the super quick answer! I wish I hadn’t spent so much time on the SSCE. 🙈

I feel like I need to add for posterity, that for the atexits actually to run on SIGTERM & SIGINT, you need to set --hook-master-start "unix_signal:15 gracefully_kill_them_all" --hook-master-start "unix_signal:2 gracefully_kill_them_all"

(master FIFO is not a good option in containers)

Um so I have another question:

Why on earth would uWSGI behave differently when I kill it with Ctrl-C and when I kill it with kill -2 <master pid>?

kill -2 <master pid>:

$ uwsgi --master --http-socket=127.0.0.1:8000 --module "mywsgi:make_app()" --lazy-apps --hook-master-start "unix_signal:2 gracefully_kill_them_all" --enable-threads
*** Starting uWSGI 2.0.15 (64bit) on [Mon Aug 14 16:21:00 2017] ***
compiled with version: 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42) on 21 July 2017 12:40:07
os: Darwin-16.7.0 Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64
nodename: alpha.local
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 8
current working directory: /Users/hynek/Projects/ssce
detected binary path: /Users/hynek/.virtualenvs/ssce/bin/uwsgi
your processes number limit is 709
your memory page size is 4096 bytes
detected max file descriptor number: 7168
lock engine: OSX spinlocks
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 127.0.0.1:8000 fd 3
Python version: 3.6.1 (default, May  4 2017, 15:25:00)  [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)]
Python main interpreter initialized at 0x7fa223002c00
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 145504 bytes (142 KB) for 1 cores
*** Operational MODE: single process ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 12800)
spawned uWSGI worker 1 (pid: 12801, cores: 1)
running "unix_signal:2 gracefully_kill_them_all" (master-start)...
thread started
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x7fa223002c00 pid: 12801 (default app)
Mon Aug 14 16:21:12 2017 - graceful shutdown triggered...
Gracefully killing worker 1 (pid: 12801)...
start clean up
msg sent
thread exiting
end clean up
worker 1 buried after 1 seconds
goodbye to uWSGI.

Ctrl-C:

$ uwsgi --master --http-socket=127.0.0.1:8000 --module "mywsgi:make_app()" --lazy-apps --hook-master-start "unix_signal:2 gracefully_kill_them_all" --enable-threads
*** Starting uWSGI 2.0.15 (64bit) on [Mon Aug 14 16:21:33 2017] ***
compiled with version: 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42) on 21 July 2017 12:40:07
os: Darwin-16.7.0 Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64
nodename: alpha.local
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 8
current working directory: /Users/hynek/Projects/ssce
detected binary path: /Users/hynek/.virtualenvs/ssce/bin/uwsgi
your processes number limit is 709
your memory page size is 4096 bytes
detected max file descriptor number: 7168
lock engine: OSX spinlocks
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 127.0.0.1:8000 fd 3
Python version: 3.6.1 (default, May  4 2017, 15:25:00)  [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)]
Python main interpreter initialized at 0x7f992e002c00
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 145504 bytes (142 KB) for 1 cores
*** Operational MODE: single process ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 12855)
spawned uWSGI worker 1 (pid: 12856, cores: 1)
running "unix_signal:2 gracefully_kill_them_all" (master-start)...
thread started
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x7f992e002c00 pid: 12856 (default app)
^CMon Aug 14 16:21:35 2017 - graceful shutdown triggered...
Gracefully killing worker 1 (pid: 12856)...
worker 1 buried after 1 seconds
goodbye to uWSGI.

How does it even know that the signal is coming from my keyboard?

And then if I run it with --threads 2 it gets even more bizarre: kill -2 will trigger the graceful shutdown, however it hangs…until I press Ctrl-C:

$ uwsgi --master --http-socket=127.0.0.1:8000 --module "mywsgi:make_app()" --lazy-apps --hook-master-start "unix_signal:2 gracefully_kill_them_all" --threads 2
*** Starting uWSGI 2.0.15 (64bit) on [Mon Aug 14 16:19:40 2017] ***
compiled with version: 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42) on 21 July 2017 12:40:07
os: Darwin-16.7.0 Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64
nodename: alpha.local
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 8
current working directory: /Users/hynek/Projects/ssce
detected binary path: /Users/hynek/.virtualenvs/ssce/bin/uwsgi
your processes number limit is 709
your memory page size is 4096 bytes
detected max file descriptor number: 7168
lock engine: OSX spinlocks
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address 127.0.0.1:8000 fd 3
Python version: 3.6.1 (default, May  4 2017, 15:25:00)  [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)]
Python main interpreter initialized at 0x7f8ce9827400
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 166080 bytes (162 KB) for 2 cores
*** Operational MODE: threaded ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 12739)
spawned uWSGI worker 1 (pid: 12740, cores: 2)
running "unix_signal:2 gracefully_kill_them_all" (master-start)...
thread started
WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x7f8ce9827400 pid: 12740 (default app)
Mon Aug 14 16:19:52 2017 - graceful shutdown triggered...
Gracefully killing worker 1 (pid: 12740)...
^Cstart clean up
msg sent
thread exiting
end clean up
worker 1 buried after 29 seconds
goodbye to uWSGI.

Could you shed some light here what is going on? Is it even possible to run WSGI application with a side-car thread?

Can you try on a Linux system ? I know it may look foolish but it is not the first time signals + pthreads result in weird behaviours on osx :(

In the mean time i can confirm that pthread cancelation does not work as expected on OSX. So you should add the option --no-threads-wait if you plan to spawn multiple threads on darwin. Basically with this option the worker is killed without waiting for threads cooperation.

OK Linux time! It’s an Ubuntu Trusty VM running in VMWare.

`--enable-threads` + `kill -2`

``` shell $ uwsgi --master --http-socket=127.0.0.1:8000 --module "mywsgi:make_app()" \ --lazy-apps --hook-master-start "unix_signal:2 gracefully_kill_them_all" \ --enable-threads *** Starting uWSGI 2.0.15 (64bit) on [Tue Aug 15 07:14:32 2017] *** compiled with version: 4.8.4 on 15 August 2017 07:11:01 os: Linux-3.13.0-110-generic #157-Ubuntu SMP Mon Feb 20 11:54:05 UTC 2017 nodename: trusty64 machine: x86_64 clock source: unix detected number of CPU cores: 2 current working directory: /home/vagrant/Projects/ssce detected binary path: /home/vagrant/.virtualenvs/ssce/bin/uwsgi !!! no internal routing support, rebuild with pcre support !!! your processes number limit is 15832 your memory page size is 4096 bytes detected max file descriptor number: 1024 lock engine: pthread robust mutexes thunder lock: disabled (you can enable it with --thunder-lock) uwsgi socket 0 bound to TCP address 127.0.0.1:8000 fd 3 Python version: 3.6.2 (default, Jul 29 2017, 00:00:00) [GCC 4.8.4] Python main interpreter initialized at 0x11a2820 python threads support enabled your server socket listen backlog is limited to 100 connections your mercy for graceful operations on workers is 60 seconds mapped 145536 bytes (142 KB) for 1 cores *** Operational MODE: single process *** *** uWSGI is running in multiple interpreter mode *** spawned uWSGI master process (pid: 12720) spawned uWSGI worker 1 (pid: 12721, cores: 1) running "unix_signal:2 gracefully_kill_them_all" (master-start)... thread started WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x11a2820 pid: 12721 (default app) Tue Aug 15 07:14:43 2017 - graceful shutdown triggered... Gracefully killing worker 1 (pid: 12721)... start clean up msg sent thread exiting end clean up worker 1 buried after 1 seconds goodbye to uWSGI. ```

Works!

`--enable-threads` + Ctrl-C

``` shell $ uwsgi --master --http-socket=127.0.0.1:8000 --module "mywsgi:make_app()" \ --lazy-apps --hook-master-start "unix_signal:2 gracefully_kill_them_all" \ --enable-threads *** Starting uWSGI 2.0.15 (64bit) on [Tue Aug 15 07:14:05 2017] *** compiled with version: 4.8.4 on 15 August 2017 07:11:01 os: Linux-3.13.0-110-generic #157-Ubuntu SMP Mon Feb 20 11:54:05 UTC 2017 nodename: trusty64 machine: x86_64 clock source: unix detected number of CPU cores: 2 current working directory: /home/vagrant/Projects/ssce detected binary path: /home/vagrant/.virtualenvs/ssce/bin/uwsgi !!! no internal routing support, rebuild with pcre support !!! your processes number limit is 15832 your memory page size is 4096 bytes detected max file descriptor number: 1024 lock engine: pthread robust mutexes thunder lock: disabled (you can enable it with --thunder-lock) uwsgi socket 0 bound to TCP address 127.0.0.1:8000 fd 3 Python version: 3.6.2 (default, Jul 29 2017, 00:00:00) [GCC 4.8.4] Python main interpreter initialized at 0xc55820 python threads support enabled your server socket listen backlog is limited to 100 connections your mercy for graceful operations on workers is 60 seconds mapped 145536 bytes (142 KB) for 1 cores *** Operational MODE: single process *** *** uWSGI is running in multiple interpreter mode *** spawned uWSGI master process (pid: 12712) spawned uWSGI worker 1 (pid: 12713, cores: 1) running "unix_signal:2 gracefully_kill_them_all" (master-start)... thread started WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0xc55820 pid: 12713 (default app) ^CTue Aug 15 07:14:09 2017 - graceful shutdown triggered... start clean up msg sent thread exiting end clean up worker 1 buried after 1 seconds goodbye to uWSGI. ```

Works!

`--threads 2` + `kill -2`

``` shell uwsgi --master --http-socket=127.0.0.1:8000 --module "mywsgi:make_app()" --lazy-apps --hook-master-start "unix_signal:2 gracefully_kill_them_all" --threads 2 *** Starting uWSGI 2.0.15 (64bit) on [Tue Aug 15 07:21:59 2017] *** compiled with version: 4.8.4 on 15 August 2017 07:11:01 os: Linux-3.13.0-110-generic #157-Ubuntu SMP Mon Feb 20 11:54:05 UTC 2017 nodename: trusty64 machine: x86_64 clock source: unix detected number of CPU cores: 2 current working directory: /home/vagrant/Projects/ssce detected binary path: /home/vagrant/.virtualenvs/ssce/bin/uwsgi !!! no internal routing support, rebuild with pcre support !!! your processes number limit is 15832 your memory page size is 4096 bytes detected max file descriptor number: 1024 lock engine: pthread robust mutexes thunder lock: disabled (you can enable it with --thunder-lock) uwsgi socket 0 bound to TCP address 127.0.0.1:8000 fd 3 Python version: 3.6.2 (default, Jul 29 2017, 00:00:00) [GCC 4.8.4] Python main interpreter initialized at 0x1622840 python threads support enabled your server socket listen backlog is limited to 100 connections your mercy for graceful operations on workers is 60 seconds mapped 166144 bytes (162 KB) for 2 cores *** Operational MODE: threaded *** *** uWSGI is running in multiple interpreter mode *** spawned uWSGI master process (pid: 12804) spawned uWSGI worker 1 (pid: 12805, cores: 2) running "unix_signal:2 gracefully_kill_them_all" (master-start)... thread started WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x1622840 pid: 12805 (default app) Tue Aug 15 07:22:04 2017 - graceful shutdown triggered... Gracefully killing worker 1 (pid: 12805)... start clean up msg sent thread exiting end clean up worker 1 buried after 1 seconds goodbye to uWSGI. ```

Seems to work but I think I had it fail occasionally. Not 100% sure tho.

`--threads 2` + Ctrl-C

``` shell $ uwsgi --master --http-socket=127.0.0.1:8000 --module "mywsgi:make_app()" \ --lazy-apps --hook-master-start "unix_signal:2 gracefully_kill_them_all" \ --threads 2 *** Starting uWSGI 2.0.15 (64bit) on [Tue Aug 15 07:12:06 2017] *** compiled with version: 4.8.4 on 15 August 2017 07:11:01 os: Linux-3.13.0-110-generic #157-Ubuntu SMP Mon Feb 20 11:54:05 UTC 2017 nodename: trusty64 machine: x86_64 clock source: unix detected number of CPU cores: 2 current working directory: /home/vagrant/Projects/ssce detected binary path: /home/vagrant/.virtualenvs/ssce/bin/uwsgi !!! no internal routing support, rebuild with pcre support !!! your processes number limit is 15832 your memory page size is 4096 bytes detected max file descriptor number: 1024 lock engine: pthread robust mutexes thunder lock: disabled (you can enable it with --thunder-lock) uwsgi socket 0 bound to TCP address 127.0.0.1:8000 fd 3 Python version: 3.6.2 (default, Jul 29 2017, 00:00:00) [GCC 4.8.4] Python main interpreter initialized at 0x1e94840 python threads support enabled your server socket listen backlog is limited to 100 connections your mercy for graceful operations on workers is 60 seconds mapped 166144 bytes (162 KB) for 2 cores *** Operational MODE: threaded *** *** uWSGI is running in multiple interpreter mode *** spawned uWSGI master process (pid: 12705) spawned uWSGI worker 1 (pid: 12706, cores: 2) running "unix_signal:2 gracefully_kill_them_all" (master-start)... thread started WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x1e94840 pid: 12706 (default app) ^CTue Aug 15 07:12:08 2017 - graceful shutdown triggered... start clean up msg sent ^Cworker 1 buried after 27 seconds goodbye to uWSGI. ```

So it apparently starts the cleanup but the join fails because it needs a second Ctrl-C.

To me, this looks awfully like some kind of race condition (who’d thought with threads :D)? Especially because very rarely it works too.

I’ve also tried sprinkle in a bunch of print(threading.enumerate()) that indicate that my thread isn't running anymore and instead there's [<_MainThread(b'uWSGIWorker1Core0', stopped 140602576090944)>, <_DummyThread(b'uWSGIWorker1Core1', started daemon 140602515154688)>]. _DummyThread!?

I don’t know if that helps you at all…here's the SSCE again if you wanna run it yourself.

``` python from __future__ import print_function import atexit import threading def make_app(): def application(env, start_response): start_response('200 OK', [('Content-Type', 'text/html')]) return [b"Hello World"] e = threading.Event() def bg_thread(): print("thread started") e.wait() print("thread exiting") print(threading.enumerate()) t = threading.Thread(target=bg_thread, name="bg-thread") t.daemon = True t.start() print(threading.enumerate()) @atexit.register def cleanup(): print("start clean up") print(threading.enumerate()) e.set() print("msg sent") print(threading.enumerate()) print(threading.current_thread(), t) t.join() print("end clean up") return application ```

Setting deamon to False does also some really funky stuff but let’s focus on one thing. ;)

Does using --no-threads-wait improves the situation ? (both in Linux and OSX)

macOS

``` shell $ uwsgi --master --http-socket=127.0.0.1:8000 --module "mywsgi:make_app()" \ --lazy-apps --hook-master-start "unix_signal:2 gracefully_kill_them_all" \ --threads 2 --no-threads-wait *** Starting uWSGI 2.0.15 (64bit) on [Tue Aug 15 08:05:20 2017] *** compiled with version: 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42) on 21 July 2017 12:40:07 os: Darwin-16.7.0 Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64 nodename: alpha.local machine: x86_64 clock source: unix pcre jit disabled detected number of CPU cores: 8 current working directory: /Users/hynek/Projects/ssce detected binary path: /Users/hynek/.virtualenvs/ssce/bin/uwsgi your processes number limit is 709 your memory page size is 4096 bytes detected max file descriptor number: 7168 lock engine: OSX spinlocks thunder lock: disabled (you can enable it with --thunder-lock) uwsgi socket 0 bound to TCP address 127.0.0.1:8000 fd 3 Python version: 3.6.1 (default, May 4 2017, 15:25:00) [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.42)] Python main interpreter initialized at 0x7f80ee017600 python threads support enabled your server socket listen backlog is limited to 100 connections your mercy for graceful operations on workers is 60 seconds mapped 166080 bytes (162 KB) for 2 cores *** Operational MODE: threaded *** *** uWSGI is running in multiple interpreter mode *** spawned uWSGI master process (pid: 31512) spawned uWSGI worker 1 (pid: 31513, cores: 2) running "unix_signal:2 gracefully_kill_them_all" (master-start)... [<_MainThread(b'uWSGIWorker1Core0', started 140736085013440)>] thread started [<_MainThread(b'uWSGIWorker1Core0', started 140736085013440)>, ] WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x7f80ee017600 pid: 31513 (default app) ^CTue Aug 15 08:05:21 2017 - graceful shutdown triggered... Gracefully killing worker 1 (pid: 31513)... worker 1 buried after 1 seconds goodbye to uWSGI. ```

Shutdown works but atexit is not executed.

Linux

``` shell uwsgi --master --http-socket=127.0.0.1:8000 --module "mywsgi:make_app()" \ --lazy-apps --hook-master-start "unix_signal:2 gracefully_kill_them_all" \ --threads 2 --no-threads-wait *** Starting uWSGI 2.0.15 (64bit) on [Tue Aug 15 08:07:07 2017] *** compiled with version: 4.8.4 on 15 August 2017 07:11:01 os: Linux-3.13.0-110-generic #157-Ubuntu SMP Mon Feb 20 11:54:05 UTC 2017 nodename: trusty64 machine: x86_64 clock source: unix detected number of CPU cores: 2 current working directory: /home/vagrant/Projects/ssce detected binary path: /home/vagrant/.virtualenvs/ssce/bin/uwsgi !!! no internal routing support, rebuild with pcre support !!! your processes number limit is 15832 your memory page size is 4096 bytes detected max file descriptor number: 1024 lock engine: pthread robust mutexes thunder lock: disabled (you can enable it with --thunder-lock) uwsgi socket 0 bound to TCP address 127.0.0.1:8000 fd 3 Python version: 3.6.2 (default, Jul 29 2017, 00:00:00) [GCC 4.8.4] Python main interpreter initialized at 0x1493890 python threads support enabled your server socket listen backlog is limited to 100 connections your mercy for graceful operations on workers is 60 seconds mapped 166144 bytes (162 KB) for 2 cores *** Operational MODE: threaded *** *** uWSGI is running in multiple interpreter mode *** spawned uWSGI master process (pid: 13787) spawned uWSGI worker 1 (pid: 13788, cores: 2) running "unix_signal:2 gracefully_kill_them_all" (master-start)... [<_MainThread(b'uWSGIWorker1Core0', started 140100408424256)>] thread started [<_MainThread(b'uWSGIWorker1Core0', started 140100408424256)>, ] WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x1493890 pid: 13788 (default app) ^CTue Aug 15 08:07:10 2017 - graceful shutdown triggered... start clean up [<_MainThread(b'uWSGIWorker1Core0', stopped 140100408424256)>, <_DummyThread(b'uWSGIWorker1Core1', started daemon 140100347750144)>] msg sent [<_MainThread(b'uWSGIWorker1Core0', stopped 140100408424256)>, <_DummyThread(b'uWSGIWorker1Core1', started daemon 140100347750144)>] Tue Aug 15 08:08:11 2017 - worker 1 (pid: 13788) is taking too much time to die...NO MERCY !!! worker 1 buried after 1 seconds goodbye to uWSGI. ```

Shutdown starts (msg sent) but hangs there, the thread doesn't seem to exist in cleanup anymore – given the prints.

Oh, this is interesting, so on the mac ctrl-c has some magic behaviour that needs to be tackled :) but on Linux there is something more strange happening as it looks like the worker is blocked in the t.join() part. Does it happen when hitting ctrl-c, when sending -2 to the master or in both cases ?

So all Linux now:

Ctrl-C

``` shell $ uwsgi --master --http-socket=127.0.0.1:8000 --module "mywsgi:make_app()" \ --lazy-apps --hook-master-start "unix_signal:2 gracefully_kill_them_all" \ --threads 2 --no-threads-wait *** Starting uWSGI 2.0.15 (64bit) on [Tue Aug 15 08:25:33 2017] *** compiled with version: 4.8.4 on 15 August 2017 07:11:01 os: Linux-3.13.0-110-generic #157-Ubuntu SMP Mon Feb 20 11:54:05 UTC 2017 nodename: trusty64 machine: x86_64 clock source: unix detected number of CPU cores: 2 current working directory: /home/vagrant/Projects/ssce detected binary path: /home/vagrant/.virtualenvs/ssce/bin/uwsgi !!! no internal routing support, rebuild with pcre support !!! your processes number limit is 15832 your memory page size is 4096 bytes detected max file descriptor number: 1024 lock engine: pthread robust mutexes thunder lock: disabled (you can enable it with --thunder-lock) uwsgi socket 0 bound to TCP address 127.0.0.1:8000 fd 3 Python version: 3.6.2 (default, Jul 29 2017, 00:00:00) [GCC 4.8.4] Python main interpreter initialized at 0x2100890 python threads support enabled your server socket listen backlog is limited to 100 connections your mercy for graceful operations on workers is 60 seconds mapped 166144 bytes (162 KB) for 2 cores *** Operational MODE: threaded *** *** uWSGI is running in multiple interpreter mode *** spawned uWSGI master process (pid: 13863) spawned uWSGI worker 1 (pid: 13864, cores: 2) running "unix_signal:2 gracefully_kill_them_all" (master-start)... [<_MainThread(b'uWSGIWorker1Core0', started 140571148867392)>] thread started [<_MainThread(b'uWSGIWorker1Core0', started 140571148867392)>, ] WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x2100890 pid: 13864 (default app) ^CTue Aug 15 08:25:34 2017 - graceful shutdown triggered... start clean up [<_MainThread(b'uWSGIWorker1Core0', stopped 140571148867392)>, <_DummyThread(b'uWSGIWorker1Core1', started daemon 140571088193280)>] msg sent [<_MainThread(b'uWSGIWorker1Core0', stopped 140571148867392)>, <_DummyThread(b'uWSGIWorker1Core1', started daemon 140571088193280)>] ^C worker 1 buried after 4 seconds goodbye to uWSGI. ```

→ the thread isn’t there and it hangs

`kill -2`

``` shell uwsgi --master --http-socket=127.0.0.1:8000 --module "mywsgi:make_app()" \ --lazy-apps --hook-master-start "unix_signal:2 gracefully_kill_them_all" \ --threads 2 --no-threads-wait *** Starting uWSGI 2.0.15 (64bit) on [Tue Aug 15 08:25:39 2017] *** compiled with version: 4.8.4 on 15 August 2017 07:11:01 os: Linux-3.13.0-110-generic #157-Ubuntu SMP Mon Feb 20 11:54:05 UTC 2017 nodename: trusty64 machine: x86_64 clock source: unix detected number of CPU cores: 2 current working directory: /home/vagrant/Projects/ssce detected binary path: /home/vagrant/.virtualenvs/ssce/bin/uwsgi !!! no internal routing support, rebuild with pcre support !!! your processes number limit is 15832 your memory page size is 4096 bytes detected max file descriptor number: 1024 lock engine: pthread robust mutexes thunder lock: disabled (you can enable it with --thunder-lock) uwsgi socket 0 bound to TCP address 127.0.0.1:8000 fd 3 Python version: 3.6.2 (default, Jul 29 2017, 00:00:00) [GCC 4.8.4] Python main interpreter initialized at 0x23a7890 python threads support enabled your server socket listen backlog is limited to 100 connections your mercy for graceful operations on workers is 60 seconds mapped 166144 bytes (162 KB) for 2 cores *** Operational MODE: threaded *** *** uWSGI is running in multiple interpreter mode *** spawned uWSGI master process (pid: 13868) spawned uWSGI worker 1 (pid: 13869, cores: 2) running "unix_signal:2 gracefully_kill_them_all" (master-start)... [<_MainThread(b'uWSGIWorker1Core0', started 139755514599232)>] thread started [<_MainThread(b'uWSGIWorker1Core0', started 139755514599232)>, ] WSGI app 0 (mountpoint='') ready in 0 seconds on interpreter 0x23a7890 pid: 13869 (default app) Tue Aug 15 08:25:46 2017 - graceful shutdown triggered... Gracefully killing worker 1 (pid: 13869)... start clean up [, <_DummyThread(b'uWSGIWorker1Core1', started daemon 139755453925120)>] msg sent thread exiting [, <_DummyThread(b'uWSGIWorker1Core1', started daemon 139755453925120)>] end clean up worker 1 buried after 1 seconds goodbye to uWSGI. ```

→ The thread is there and it works.

I gotta run now, so I suggest you play with my SSCE I’ve posted before, it has no dependencies and works on both Python 2 and Python 3. :)

JFTR, I remembered why Ctrl-C and kill -2 work differently: doing a Ctrl-C fires a SIGINT to all foreground processes. That seems to confuse things.

I recently working on a nginx+uwsgi+flask+golang web service. Before i add some golang shared library for python ,it all work fine.But if i call golang shared library in my python script,it stuck without any error, just no response from golang.(it works fine if i call golang shared library with python interpreter) after i add lazy-apps and --no-threads-wait , the stucking problem fixed!!! I think it maybe some problem with gorotuine,but i not sure the main reason. Can anyone explain this,or maybe give me some keyword that i can try to find

unbit / uwsgi

--master, atexit, and threads #1609

`--enable-threads` + `kill -2`

`--enable-threads` + Ctrl-C

`--threads 2` + `kill -2`

`--threads 2` + Ctrl-C

macOS

Linux

Ctrl-C

`kill -2`

unbit / uwsgi

--master, atexit, and threads #1609

--enable-threads + kill -2

--enable-threads + Ctrl-C

--threads 2 + kill -2

--threads 2 + Ctrl-C

macOS

Linux

Ctrl-C

kill -2

`--enable-threads` + `kill -2`

`--enable-threads` + Ctrl-C

`--threads 2` + `kill -2`

`--threads 2` + Ctrl-C

`kill -2`