uWSGI doesn't respond after touch-reload

[tarunlalwani]

I have a weird issue with uWSGI not reloading the environment once i touch the files. Below are my ini files

# main-server.ini
[uwsgi]
emperor = /root/automation/serverui/vassals/
logto = /var/log/docker/mainserver.log
die-on-term = true
py-autoreload = 0
vassal-set = processes=2
vassal-set = workers=2
vassal-set = vaccum=true
vassal-set = master=true
vassal-set = py-autoreload=0
vassal-set = thunder-lock = true

# dockercomposeui.ini
[uwsgi]
module = main:app
base = %d/../docker-compose-ui/
touch-reload = /root/automation/serverui/vassals/dockercomposeui.ini
http = 127.0.0.1:5000
#protocol = http
pidfile=/tmp/%n.pid
daemonize=/var/log/uwsgi/%n.log

chdir = /root/automation/serverui/docker-compose-ui/
env = PYTHON_PATH=$(base):.:$(PYTHON_PATH)
env = DOCKER_COMPOSE_UI_YML_PATH=/root/automation/serverui/projects/

#serverui.ini
[uwsgi]
module = serverui.uwsgi:app
pidfile=/tmp/%n.pid
daemonize=/var/log/uwsgi/%n.log
logto = /var/log/docker/dockerserverui.log
base = %d
touch-reload = /root/automation/serverui/vassals/serverui.ini
http = 0.0.0.0:8010
chdir = /root/automation/
env = PYTHON_PATH=$(base)/../:.:$(PYTHON_PATH)
wsgi-file = %(base)wsgi.py

[root@localhost serverui]# uwsgi --version
2.0.13.1

[root@localhost serverui]# uname -a
Linux localhost.localdomain 3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Below are the log entries

*** Starting uWSGI 2.0.13.1 (64bit) on [Wed May 25 13:03:02 2016] ***
compiled with version: 4.8.5 20150623 (Red Hat 4.8.5-4) on 22 May 2016 21:48:14
os: Linux-3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015
nodename: localhost.localdomain
machine: x86_64
clock source: unix
detected number of CPU cores: 32
current working directory: /root/automation/serverui/vassals
writing pidfile to /tmp/serverui.pid
detected binary path: /usr/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
chdir() to /root/automation/
your processes number limit is 514507
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uWSGI http bound on 0.0.0.0:8010 fd 4
uwsgi socket 0 bound to TCP address 127.0.0.1:39377 (port auto-assigned) fd 3
Python version: 2.7.5 (default, Nov 20 2015, 02:00:19)  [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)]
Python main interpreter initialized at 0x22d7ef0
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 218280 bytes (213 KB) for 2 cores
*** Operational MODE: preforking ***
WSGI app 0 (mountpoint='') ready in 1 seconds on interpreter 0x22d7ef0 pid: 12566 (default app)
mountpoint  already configured. skip.
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 12566)
spawned uWSGI worker 1 (pid: 12577, cores: 1)
spawned uWSGI worker 2 (pid: 12578, cores: 1)
spawned uWSGI http 1 (pid: 12579)
[pid: 12578|app: 0|req: 1/1] 10.100.81.185 () {42 vars in 1160 bytes} [Wed May 25 13:03:10 2016] GET /process => generated 11607 bytes in 131 msecs (HTTP/1.1 200) 3 headers in 420 bytes (1 switches on core 0)
announcing my loyalty to the Emperor...
[pid: 12578|app: 0|req: 2/2] 10.100.81.185 () {42 vars in 1158 bytes} [Wed May 25 13:03:14 2016] GET / => generated 8237 bytes in 21 msecs (HTTP/1.1 200) 3 headers in 419 bytes (1 switches on core 0)
[pid: 12578|app: 0|req: 3/3] 10.100.81.185 () {42 vars in 1175 bytes} [Wed May 25 13:03:16 2016] GET /dockers/pp90 => generated 29535 bytes in 15372 msecs (HTTP/1.1 200) 3 headers in 420 bytes (1 switches on core 0)
Wed May 25 13:03:59 2016 - *** /root/automation/serverui/vassals/serverui.ini has been touched... grace them all !!! ***
gateway "uWSGI http 1" has been buried (pid: 12579)
...gracefully killing workers...
Gracefully killing worker 1 (pid: 12577)...
Gracefully killing worker 2 (pid: 12578)...
worker 1 buried after 1 seconds
worker 2 buried after 1 seconds
binary reloading uWSGI...
chdir() to /root/automation/serverui/vassals
closing all non-uwsgi socket fds > 2 (max_fd = 1024)...
found fd 3 mapped to socket 0 (127.0.0.1:39377)
running /usr/bin/uwsgi
*** has_emperor mode detected (fd: 7) ***
[uWSGI] getting INI configuration from serverui.ini

# mainserver.log
*** Starting uWSGI 2.0.13.1 (64bit) on [Wed May 25 13:03:02 2016] ***
compiled with version: 4.8.5 20150623 (Red Hat 4.8.5-4) on 22 May 2016 21:48:14
os: Linux-3.10.0-327.el7.x86_64 #1 SMP Thu Nov 19 22:10:57 UTC 2015
nodename: localhost.localdomain
machine: x86_64
clock source: unix
detected number of CPU cores: 32
current working directory: /root/payu-automation/serverui
detected binary path: /usr/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
your processes number limit is 514507
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
*** starting uWSGI Emperor ***
Python version: 2.7.5 (default, Nov 20 2015, 02:00:19)  [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)]
*** has_emperor mode detected (fd: 6) ***
*** has_emperor mode detected (fd: 7) ***
[uWSGI] getting INI configuration from dockercomposeui.ini
[uWSGI] getting INI configuration from serverui.ini
Python main interpreter initialized at 0x9444e0
python threads support enabled
your mercy for graceful operations on workers is 60 seconds
*** Operational MODE: no-workers ***
spawned uWSGI master process (pid: 12563)
Wed May 25 13:03:03 2016 - [emperor] vassal serverui.ini has been spawned
Wed May 25 13:03:03 2016 - [emperor] vassal serverui.ini is ready to accept requests
Wed May 25 13:03:03 2016 - [emperor] vassal dockercomposeui.ini has been spawned
Wed May 25 13:03:03 2016 - [emperor] vassal dockercomposeui.ini is ready to accept requests
Wed May 25 13:03:10 2016 - [emperor] vassal serverui.ini is now loyal
Wed May 25 13:03:16 2016 - [emperor] vassal dockercomposeui.ini is now loyal
Wed May 25 13:03:16 2016 - [emperor] vassal dockercomposeui.ini is now loyal
Wed May 25 13:04:01 2016 - [emperor] reload the uwsgi instance dockercomposeui.ini
Wed May 25 13:04:01 2016 - [emperor] reload the uwsgi instance serverui.ini
[emperor] *** RAGNAROK EVOKED ***
SIGINT/SIGQUIT received...killing workers...
Wed May 25 13:08:24 2016 - [emperor] stop the uwsgi instance dockercomposeui.ini
waiting for Emperor death...
Wed May 25 13:08:24 2016 - [emperor] stop the uwsgi instance serverui.ini
[emperor] *** RAGNAROK ALREADY EVOKED (mercyless in 30 seconds)***
waiting for Emperor death...
Wed May 25 13:08:25 2016 - [emperor] removed uwsgi instance dockercomposeui.ini
Wed May 25 13:08:25 2016 - [emperor] removed uwsgi instance serverui.ini
waiting for Emperor death...
Wed May 25 13:08:26 2016 - The Emperor is buried.
The Emperor has been buried (pid: 12564)
goodbye to uWSGI.

[xrmx]

@tarunlalwani from the logs it looks like someone killed the emperor. Can you provide a smaller reproducer and the list of commands you have done please?

[tarunlalwani]

@xrmx, I only killed the emperor. After checking that it is not working. The only two commands i executed is below

$ uwsgi --ini mainserver.ini

$ touch vassals/*

I am able to workaround the issue now by setting master=false for both the vassals. But all example on the net shows setting master=true in the vassals ini file. Now I am not sure what's wrong

[tarunlalwani]

@xrmx , thoughts?

[olitomas]

@tarunlalwani did you figure this out ? If so, do you remember what was wrong ?

[tarunlalwani]

@olitomas , for me setting master=false in the vassals had worked fine, so I used it as it is