unbit / uwsgi

uWSGI application server container
http://projects.unbit.it/uwsgi
Other
3.45k stars 691 forks source link

uWSGI "SIGNAL QUEUE IS FULL" errors #2123

Closed LeeKevin closed 4 years ago

LeeKevin commented 4 years ago

I have a Django application running with uWSGI.

Sometimes, when restarting the uWSGI service, I get a bunch of *** SIGNAL QUEUE IS FULL: buffer size 212992 bytes (you can tune it with --signal-bufsize) *** errors.

When that happens, I'm unable to access the Django app. Restarting uWSGI again usually works.

Any ideas what happening here?

On startup of my application, I have a bunch of crons and timers that are initialized, like:

@uwsgidecorators.timer(30)
def send_queued_mail(*args, **kwargs):

  ...

The full output of the wsgi process is below:

chdir() to /etc/uwsgi/vassals
closing all non-uwsgi socket fds > 2 (max_fd = 1024)...
found fd 3 mapped to socket 0 (/var/run/uwsgi/MY_APP.sock)
running /usr/local/bin/uwsgi
*** has_emperor mode detected (fd: 7) ***
[uWSGI] getting INI configuration from MY_APP.ini
*** Starting uWSGI 2.0.17.1 (64bit) on [Thu Feb  6 22:22:18 2020] ***
compiled with version: 4.8.4 on 07 August 2018 17:29:43
os: Linux-4.2.0-42-generic #49~14.04.1-Ubuntu SMP Wed Jun 29 20:22:11 UTC 2016
nodename: spuzzum
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 4
current working directory: /etc/uwsgi/vassals
detected binary path: /usr/local/bin/uwsgi
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
chdir() to /opt/MY_APP
your processes number limit is 128614
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 inherited UNIX address /var/run/uwsgi/MY_APP.sock fd 3
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
Python version: 3.6.7 (default, Oct 21 2018, 00:00:00)  [GCC 4.8.4]
Set PythonHome to /opt/MY_APP/venv/MY_APP-3.6
Python main interpreter initialized at 0x24b1a00
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 609552 bytes (595 KB) for 5 cores
*** Operational MODE: preforking ***
*** Starting uWSGI 2.0.17.1 (64bit) on [Thu Feb  6 22:22:21 2020] ***
compiled with version: 4.8.4 on 07 August 2018 17:29:43
os: Linux-4.2.0-42-generic #49~14.04.1-Ubuntu SMP Wed Jun 29 20:22:11 UTC 2016
nodename: spuzzum
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 4
current working directory: /etc/uwsgi/vassals
detected binary path: /usr/local/bin/uwsgi
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
chdir() to /opt/MY_APP
your processes number limit is 128614
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 inherited UNIX address /var/run/uwsgi/MY_APP.sock fd 3
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
Python version: 3.6.7 (default, Oct 21 2018, 00:00:00)  [GCC 4.8.4]
Set PythonHome to /opt/MY_APP/venv/MY_APP-3.6
Python main interpreter initialized at 0x24b1a00
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 609552 bytes (595 KB) for 5 cores
*** Operational MODE: preforking ***
*** Starting uWSGI 2.0.17.1 (64bit) on [Thu Feb  6 22:22:21 2020] ***
compiled with version: 4.8.4 on 07 August 2018 17:29:43
os: Linux-4.2.0-42-generic #49~14.04.1-Ubuntu SMP Wed Jun 29 20:22:11 UTC 2016
nodename: spuzzum
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 4
current working directory: /etc/uwsgi/vassals
detected binary path: /usr/local/bin/uwsgi
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
chdir() to /opt/MY_APP
your processes number limit is 128614
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to UNIX address /var/run/uwsgi/MY_APP.sock fd 3
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
Python version: 3.6.7 (default, Oct 21 2018, 00:00:00)  [GCC 4.8.4]
Set PythonHome to /opt/MY_APP/venv/MY_APP-3.6
Python main interpreter initialized at 0x10899e0
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
python threads support enabled
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 609552 bytes (595 KB) for 5 cores
*** Operational MODE: preforking ***
*** has_emperor mode detected (fd: 7) ***
unable to load configuration from from multiprocessing.semaphore_tracker import main;main(17)
[uwsgi-signal] signum 0 registered (wid: 0 modifier1: 0 target: default, any worker)
[uwsgi-signal] signum 1 registered (wid: 0 modifier1: 0 target: default, any worker)
[uwsgi-signal] signum 2 registered (wid: 0 modifier1: 0 target: default, any worker)
[uwsgi-signal] signum 3 registered (wid: 0 modifier1: 0 target: default, any worker)
[uwsgi-signal] signum 4 registered (wid: 0 modifier1: 0 target: default, any worker)
[uwsgi-signal] signum 5 registered (wid: 0 modifier1: 0 target: default, any worker)
[uwsgi-signal] signum 6 registered (wid: 0 modifier1: 0 target: default, any worker)
[uwsgi-signal] signum 7 registered (wid: 0 modifier1: 0 target: default, any worker)
[uwsgi-signal] signum 8 registered (wid: 0 modifier1: 0 target: default, any worker)
[uwsgi-signal] signum 9 registered (wid: 0 modifier1: 0 target: default, any worker)
[uwsgi-signal] signum 10 registered (wid: 0 modifier1: 0 target: default, any worker)
[uwsgi-signal] signum 11 registered (wid: 0 modifier1: 0 target: default, any worker)
[uwsgi-signal] signum 12 registered (wid: 0 modifier1: 0 target: default, any worker)
[uwsgi-signal] signum 13 registered (wid: 0 modifier1: 0 target: default, any worker)
[uwsgi-signal] signum 14 registered (wid: 0 modifier1: 0 target: default, any worker)
[uwsgi-signal] signum 15 registered (wid: 0 modifier1: 0 target: default, any worker)
[uwsgi-signal] signum 16 registered (wid: 0 modifier1: 0 target: default, any worker)
WSGI app 0 (mountpoint='') ready in 11 seconds on interpreter 0x10899e0 pid: 8515 (default app)
uWSGI running as root, you can use --uid/--gid/--chroot options
*** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 8515)
spawned uWSGI worker 1 (pid: 8548, cores: 1)
spawned uWSGI worker 2 (pid: 8549, cores: 1)
spawned uWSGI worker 3 (pid: 8550, cores: 1)
spawned uWSGI worker 4 (pid: 8551, cores: 1)
spawned uWSGI worker 5 (pid: 8552, cores: 1)
*** Stats server enabled on /tmp/MY_APP-stats.socket fd: 23 ***
subprocess 8540 exited with code 1
*** SIGNAL QUEUE IS FULL: buffer size 212992 bytes (you can tune it with --signal-bufsize) ***
could not deliver signal 14 to workers pool
*** SIGNAL QUEUE IS FULL: buffer size 212992 bytes (you can tune it with --signal-bufsize) ***
could not deliver signal 0 to workers pool
*** SIGNAL QUEUE IS FULL: buffer size 212992 bytes (you can tune it with --signal-bufsize) ***
could not deliver signal 0 to workers pool
*** SIGNAL QUEUE IS FULL: buffer size 212992 bytes (you can tune it with --signal-bufsize) ***
could not deliver signal 13 to workers pool
*** SIGNAL QUEUE IS FULL: buffer size 212992 bytes (you can tune it with --signal-bufsize) ***
could not deliver signal 14 to workers pool
*** SIGNAL QUEUE IS FULL: buffer size 212992 bytes (you can tune it with --signal-bufsize) ***
xrmx commented 4 years ago

Any chance you are adding a the very same timer multiple times at runtime? Is the decorated function imported in a separated file from your django app via a settings like import = <yourmodule>?

Also looks like you are doing something fishy there:

unable to load configuration from from multiprocessing.semaphore_tracker import main;main(17)
LeeKevin commented 4 years ago

Any chance you are adding a the very same timer multiple times at runtime?

Good point. I will test for this.

Is the decorated function imported in a separated file from your django app via a settings like import = ?

The decorated functions are declared directly in the my wsgi.py file where the django app is created:

wsgi.py

import os
import traceback

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "MY_APP.settings")

from django.conf import settings
from django.utils import timezone

if settings.DEBUG and settings.DEVELOPMENT:
    from ws4redis.django_runserver import application

    application = application
else:
    from django.core.wsgi import get_wsgi_application

    application = get_wsgi_application()

try:
    import uwsgidecorators
    from django.core.management import call_command

    # Send mail
    @uwsgidecorators.lock
    @uwsgidecorators.timer(30)
    def send_queued_mail(*args, **kwargs):
        ...

...

Is that bad practice? Would appreciate input on that.

Also looks like you are doing something fishy [with multiprocessing]

I looked through my app and I see that the multiprocessing module is imported once in "deeper" part of the application. It's used in a function to check the number of cpus. Any potential issues with that import?

xrmx commented 4 years ago

The documentations suggests to use a separeted file: https://uwsgi-docs.readthedocs.io/en/latest/PythonDecorators.html

LeeKevin commented 4 years ago

Excellent, that's the kind of thing I was looking for.

I will fix my application and close the issue once all looks good.

ataotaotao commented 4 years ago

Could you tell me how to solve this problem? It just drives me crazy!