nginx+uwsgi 504 Bad Gateway Time-Out

[gamilwcy]

Hello, I have encountered the following issue. After several days of testing, I have come to the conclusion and the logs are as follows. I am not sure if it is my setup error or the original Luci ssl nginx bug. Please help me analyze it. Thank you very much.

Maintainer: @\<github-user> Environment: (x86_64, x86_64, LEDE for Lean)

Description:

When I used Docker to pull the image, onlyOffice/documentserver: latest, and downloaded it for 1 minute, I noticed that it
 happened to be 60 seconds, and suddenly returned 504. Then, I switched to uhtpd and did the same operation without this 
problem. I tried to set many parameters for nginx and modify the parameters for uwsgi, but to no avail, I returned an error very
 accurately at 60 seconds. But if I remove smaller images such as nginx, this problem will not occur.
The following are the logs of nginx and logread:

The nginx logs:

2024/02/13 14:48:14 [error] 3132#0: *22 upstream prematurely closed connection while reading response header from upstream, client: 192.168.1.157, server: _lan, request: "POST /cgi-bin/luci/admin/docker/images HTTP/1.1", upstream: "uwsgi://unix:////var/run/luci-webui.socket:", host: "192.168.1.229", referrer: "https://192.168.1.229/cgi-bin/luci/admin/docker/images"

The uwsgi logs

/etc/uwsgi/vassals/luci-webui.ini is ready to accept requests Tue Feb 13 14:47:23 2024 daemon.err uwsgi[31940]: Tue Feb 13 14:47:23 2024 - [emperor] vassal /etc/uwsgi/vassals/luci-webui.ini is now loyal Tue Feb 13 14:48:14 2024 daemon.info uwsgi-luci: invalid CGI response !!! Tue Feb 13 14:48:14 2024 daemon.err dockerd[13056]: time="2024-02-13T06:48:14.918256230Z" level=error msg="Not continuing with pull after error: context canceled"

Run Env

• LEDE(LEAN) of OpenWRT
• NGINX: 1.21.3
• Packege: luci-ssl-nginx uwsgi

nginx configuration

    server_names_hash_bucket_size 512;
    client_header_buffer_size 32k;
    large_client_header_buffers 4 32k;
client_max_body_size 8g;

gzip on;
gzip_vary on;
gzip_proxied any;

    fastcgi_connect_timeout 300s;
    fastcgi_send_timeout 300s;
    fastcgi_read_timeout 300s;
    fastcgi_buffer_size 256k;
    fastcgi_buffers 16 256k;
    fastcgi_busy_buffers_size 512k;
    fastcgi_temp_file_write_size 512k;
            fastcgi_intercept_errors on;

root /www;

server { #see uci show 'nginx._lan'
    listen 443 ssl default_server;
    listen [::]:443 ssl default_server;
    server_name _lan;
    include restrict_locally;
    include conf.d/*.locations;
    ssl_certificate /etc/nginx/conf.d/_lan.crt;
    ssl_certificate_key /etc/nginx/conf.d/_lan.key;
    ssl_session_cache shared:SSL:32k;
    ssl_session_timeout 64m;
    access_log off; # logd openwrt;
}

server { #see uci show 'nginx._redirect2ssl'
    listen 80;
    listen [::]:80;
    server_name _redirect2ssl;
    return 302 https://$host$request_uri;
}

location /cgi-bin/luci {
        index  index.html;
        include uwsgi_params;
        uwsgi_param SERVER_ADDR $server_addr;
        uwsgi_modifier1 9;
        uwsgi_read_timeout 1800;
        uwsgi_send_timeout 300;
        proxy_read_timeout 75s;
        proxy_send_timeout 75s;
        proxy_connect_timeout 75s;
        uwsgi_socket_keepalive on;
        uwsgi_force_ranges on;
        uwsgi_connect_timeout 300s;
        uwsgi_ignore_client_abort on;
        uwsgi_next_upstream_timeout 120s;
        uwsgi_next_upstream_tries 10;
        uwsgi_max_temp_file_size 4096m;
        uwsgi_pass unix:////var/run/luci-webui.socket;
}
location ~ /cgi-bin/cgi-(backup|download|upload|exec) {
        include uwsgi_params;
        uwsgi_param SERVER_ADDR $server_addr;
        uwsgi_modifier1 9;
        uwsgi_read_timeout 1800;
        uwsgi_send_timeout 300;
        proxy_read_timeout 75;
        proxy_send_timeout 75s;
        proxy_connect_timeout 75s;
        uwsgi_socket_keepalive on;
        uwsgi_force_ranges on;
        uwsgi_connect_timeout 120s;
        uwsgi_ignore_client_abort on;
        uwsgi_next_upstream_timeout 120s;
        uwsgi_next_upstream_tries 10;
        uwsgi_max_temp_file_size 4096m;
        uwsgi_pass unix:////var/run/luci-cgi_io.socket;
}

location /luci-static {
        error_log stderr crit;
}

location /ubus {
        ubus_interpreter;
        ubus_socket_path /var/run/ubus/ubus.sock;
        ubus_parallel_req 2;
}

Please pay attention to the details. I use a stopwatch to time every 60 seconds, and it returns 504

[brada4]

Please report it where you got (rather ancient) packages from: https://github.com/coolsnowwolf/packages/issues

[gamilwcy]

Please report it where you got (rather ancient) packages from: https://github.com/coolsnowwolf/packages/issues

OK,I can verify if the latest version of OpenWrt has this issue. Thks.

[brada4]

Obvious cancellation comes from the side of uwsgi, configured in some INI file.

[gamilwcy]

Obvious cancellation comes from the side of uwsgi, configured in some INI file.

Please help me see which configuration option needs to be modified. Thank you very much.

This is my uwsgi configurations

/etc/uwsgi/emperor.ini

[uwsgi] strict = true pidfile = /var/run/uwsgi.pid emperor = /etc/uwsgi/vassals/*.ini early-emperor = true vacuum = true emperor-on-demand-directory = /var/run/ emperor-required-heartbeat = 99 vassal-set = die-on-idle=true

/etc/uwsgi/vassals/luci-webui.ini

[uwsgi] strict = true if-not-env = UWSGI_EMPEROR_FD socket = /var/run/luci-webui.socket chmod-socket = 666 cheap = true end-if = plugin = cgi cgi-mode = true cgi = /www/ chdir = /usr/lib/lua/luci/ buffer-size = 10000 reload-mercy = 8 max-requests = 2000 limit-as = 600000 reload-on-as = 256 reload-on-rss = 192 enable-threads = true post-buffering = 8192 socket-timeout = 300 thunder-lock = true plugin = syslog logger = luci syslog:uwsgi-luci ; the regular expression leaves for successful de/activation only one line each: log-route = luci ^(?!... Starting uWSGI |compiled with version: |os: Linux|nodename: |machine: |clock source: |pcre jit |detected number of CPU cores: |current working directory: |detected binary path: |uWSGI running as root, you can use |... WARNING: you are running uWSGI as root |chdir.. to |your processes number limit is |limiting address space of processes...|your process address space limit is |your memory page size is |detected max file descriptor number: |lock engine: |thunder lock: |uwsgi socket |your server socket listen backlog is limited to |your mercy for graceful operations on workers is |mapped . bytes |... Operational MODE: |initialized CGI path: |... no app loaded. going in full dynamic mode ...|... uWSGI is running in multiple interpreter mode ...|spawned uWSGI worker |announcing my loyalty to the Emperor...|workers have been inactive for more than |SIGINT/SIGQUIT received...killing workers...|worker . buried |goodbye to uWSGI.|...gracefully killing workers...|Gracefully killing worker|worker .* killed successfully) disable-logging = true req-logger = syslog:uwsgi-luci log-format=%(method) %(uri) => return %(status) (%(rsize) bytes in %(msecs) ms) threads = 3 processes = 3 cheaper-algo = spare cheaper = 1 cheaper-initial = 1 cheaper-step = 1 master = true idle = 360

[brada4]

uwsgi has documentation on idlekill timeouts. Enormous timeouts are not justifiable for general case.