dstndstn
commented
2 years ago Sorry, forgot to say: upon restarting uwsgi (service nova-uwsgi restart), things work fine again.
And with --show-config, the log file starts with:
*** Starting uWSGI 2.0.20 (64bit) on [Tue May 31 14:05:48 2022] ***
compiled with version: 9.3.0 on 18 January 2022 15:56:23
os: Linux-5.4.0-113-generic #127-Ubuntu SMP Wed May 18 14:30:56 UTC 2022
nodename: nova
machine: x86_64
clock source: unix
pcre jit disabled
detected number of CPU cores: 16
current working directory: /home/nova/astrometry/net
detected binary path: /usr/local/bin/uwsgi
your processes number limit is 482928
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to TCP address :3030 fd 4
Python version: 3.8.10 (default, Mar 15 2022, 12:22:08) [GCC 9.4.0]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x5557afda4680
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 656136 bytes (640 KB) for 8 cores
*** Operational MODE: preforking ***
Path: /home/nova/astrometry
Logging to /data/nova/nova.log
WSGI app 0 (mountpoint='') ready in 17 seconds on interpreter 0x5557afda4680 pid: 2597129 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 2597129)
spawned uWSGI worker 1 (pid: 2597149, cores: 1)
spawned uWSGI worker 2 (pid: 2597150, cores: 1)
spawned uWSGI worker 3 (pid: 2597151, cores: 1)
spawned uWSGI worker 4 (pid: 2597152, cores: 1)
spawned uWSGI worker 5 (pid: 2597153, cores: 1)
spawned uWSGI worker 6 (pid: 2597154, cores: 1)
spawned uWSGI worker 7 (pid: 2597155, cores: 1)
spawned uWSGI worker 8 (pid: 2597156, cores: 1)
*** Stats server enabled on 127.0.0.1:1717 fd: 24 ***
[pid: 2597152|app: 0|req: 1/2] 99.250.167.114 () {76 vars in 1374 bytes} [Tue May 31 14:06:05 2022] GET / => generated 14924 bytes in 1398 msecs (HTTP/1.1 200) 7 headers in 223 bytes (1 switches on core 0
)and I guess I should mention that this is a Python/Django site.
Thanks!
Hi,
I'm running an Ubuntu 20.04 on an OpenStack VM, Intel broadwell CPU. Apache 2.4. Uwsgi 2.0.20.
uwsgi run (in a script that is run by systemd, as a normal user):
After a day or two on my ~busy site (few requests/second), I am getting:
repeated. In
top, I don't see any uWSGI processes using CPU, and I think they're in state "S" in ps. It doesn't look like it's busy doing anything, it looks like it's just stalled.Previously, I was getting dmesg entries about CPU soft lock-ups -- flaky hardware underlying the VM? -- but in today's edition of this problem I don't think I see that.
Looking in the Apache logs, things seem normal leading up to this time, and then I start seeing a bunch of requests that end with http status 500 after 300 seconds.
I'm feeling lost about what to look at to figure out what is going on. I'm still not 100% sure it's not hardware-related. One odd thing I'm seeing is that the process ids are large, like the current uwsgi process is pid 2618919. Maybe that's just life on a working server, but it seemed unusual.
Any hints on things to look into, stats to collect, logging to do, things to investigate next time this happens, would be very much appreciated!
Thank you, dustin