uwsgi-cgi: Stalls / deadlocks under heavy load

openwrt / packages

Community maintained packages for OpenWrt. Documentation for submitting pull requests is in CONTRIBUTING.md

GNU General Public License v2.0

3.98k stars 3.47k forks source link

uwsgi-cgi: Stalls / deadlocks under heavy load #7250

Closed rossbcan closed 4 years ago

rossbcan commented 5 years ago

Hi;

This may be more of a default configuration issue. I encountered deadlock under heavy load, discussed here:

https://forum.openwrt.org/t/solved-default-postinst-hangs/22946/8

solved (or at least reduced) by "limit-as = 1000" (from 200)

I am using OpenWrt X86_64, SMP (2 processors), 4GB RAM, 4GB VMEM (swap).

Seeking advice on how to set other related parameters in uwsgi.conf to tune performance, minimise stalls such as:

reload-on-as reload-on-rss

Also, a lot of syslog clutter. Suggest default conf be log to file, eg: logger = file:/var/log/uwsgi.log

Regards; Bill

neheb commented 4 years ago

ping @peter-stadler

I don't know how valid this issue is anymore. uwsgi-cgi seems to have been replaced.

peter-stadler commented 4 years ago

It is right that uwsgi-cgi is replaced by a modular build of uwsgi and the conf files changed a bit to handle cgi_io.

But, there is still limit-as = 200 in the luci configuration files /etc/uwsgi/luci-{webui,cgi_io}.ini. From my side we could change that, what do you think @Ansuel? And would it be good to change the other parameters, too?

We changed the default logging for the new package, so there should be less clutter. We could also log to a file by using uwsgi-logfile-plugin instead of uwsgi-syslog-plugin ...

rossbcan commented 4 years ago

have gone with limit-as = 1000 no issues since then

Ansuel commented 4 years ago

@peter-stadler should we change the log to write it to a file like we do for nginx ?

peter-stadler commented 4 years ago

I think that is not needed, I would rather reduce the logging of Nginx (to log only errors) and include it into syslog. For uWSGI we have:

The emperor logs about 20 lines on (re)start, but that should be really really seldom.
For vassals the emperor logs 5+3 lines when started+stopped; imo that is OK depending on how often that happens; For on-demand vassals (like luci-*) this is also influenced by the idle parameter.
The vassals can decide how much they log and where; For LuCI that is just the login. For debugging one can activate more verbose logging, e.g., by setting disable-logging = false in luci-*.ini

Personally, I am not using LuCI that much that it would need own files. But, feel free to change it. I am preparing the new PR for Nginx (to use the conf.d directory) and would match the logging behavior (files/syslog) to uWSGI.

neheb commented 4 years ago

Relevant PR was merged. Closing.