search

NginxProxyManager / nginx-proxy-manager

Docker container for managing Nginx proxy hosts with a simple, powerful interface
https://nginxproxymanager.com
MIT License
22.25k stars 2.57k forks source link

"Random" 502 errors #2608

Open ValentinAUCLERC opened 1 year ago

ValentinAUCLERC commented 1 year ago

Checklist

Describe the bug I'm using NPM with multiple images (portainer, httpd ...) and I made the NPM docker-compose join each sub-network for my other docker-compose projects.

When i'm accessing directly, let's say my httpd project with exposed port, no problem, even if i'm mashing F5 button. When i'm doing the same through the NPM proxy, I have "random" 502 errors, I would say nearly 5-10% of the time, with the same url.

Meanwhile no problem doing the same with the direct access.

Nginx Proxy Manager Version v.2.9.19

To Reproduce

Operating System Docker on Ubuntu Server

ValentinAUCLERC commented 1 year ago

Here is my docker-compose.yml for NPM

version: '2.4'
services:
  app:
    container_name: nginxproxymanager
    image: 'jc21/nginx-proxy-manager:latest'
    restart: unless-stopped
    environment:
      ENABLE_IPV6: true
    ports:
      - '80:80'
      - '85:81'
      - '443:443'
    volumes:
      - ./data:/data
      - ./letsencrypt:/etc/letsencrypt
    networks:
      - p1
      - p2
      - p3
networks:
  p1:
    external: true
  p2:
    external: true
  p3:
    external: true
ValentinAUCLERC commented 1 year ago

More info :

I just looked at my data/logs/proxy-host-7_error.log and got this message :

023/02/12 21:16:09 [error] 782#782: *176817 [myhost] could not be resolved (3: Host not found) But at the next f5... it finds it

LukeSkywalker993 commented 1 year ago

Hello, I also have the random problem with the 502 errors. I have that since the last update on all proxies. I can just press F5 a few times or wait. That's pretty annoying :-(

Before the update everything was working fine!!

Errlog:

2023/02/12 21:52:37 [error] 681#681: 1252331 upstream prematurely closed connection while reading response header from upstream, client: 172.21.0.1, server: symcon.domain.de, request: "POST /hook/ipsviewconnect/api/ HTTP/1.1", upstream: "https://192.168.26.5:3778/hook/ipsviewconnect/api/", host: "symcon.domain.de" 2023/02/12 21:52:37 [error] 682#682: 1252334 peer closed connection in SSL handshake while SSL handshaking to upstream, client: 172.21.0.1, server: symcon.domain.de, request: "POST /hook/ipsviewconnect/api/ HTTP/1.1", upstream: "https://192.168.26.5:3778/hook/ipsviewconnect/api/", host: "symcon.domain.de" 2023/02/12 21:52:37 [error] 682#682: 1252333 upstream prematurely closed connection while reading response header from upstream, client: 172.21.0.1, server: symcon.domain.de, request: "POST /hook/ipsviewconnect/api/ HTTP/1.1", upstream: "https://192.168.26.5:3778/hook/ipsviewconnect/api/", host: "symcon.domain.de" 2023/02/12 21:52:37 [error] 681#681: 1252338 peer closed connection in SSL handshake while SSL handshaking to upstream, client: 172.21.0.1, server: symcon.domain.de, request: "POST /hook/ipsviewconnect/api/ HTTP/1.1", upstream: "https://192.168.26.5:3778/hook/ipsviewconnect/api/", host: "symcon.domain.de" 2023/02/12 21:52:37 [error] 682#682: 1252332 upstream prematurely closed connection while reading response header from upstream, client: 172.21.0.1, server: symcon.domain.de, request: "POST /hook/ipsviewconnect/api/ HTTP/1.1", upstream: "https://192.168.26.5:3778/hook/ipsviewconnect/api/", host: "symcon.domain.de" 2023/02/12 21:52:37 [error] 682#682: 1252335 upstream prematurely closed connection while reading response header from upstream, client: 172.21.0.1, server: symcon.domain.de, request: "POST /hook/ipsviewconnect/api/ HTTP/1.1", upstream: "https://192.168.26.5:3778/hook/ipsviewconnect/api/", host: "symcon.domain.de" 2023/02/12 21:52:37 [error] 682#682: 1252336 peer closed connection in SSL handshake while SSL handshaking to upstream, client: 172.21.0.1, server: symcon.domain.de, request: "POST /hook/ipsviewconnect/api/ HTTP/1.1", upstream: "https://192.168.26.5:3778/hook/ipsviewconnect/api/", host: "symcon.domain.de" 2023/02/12 21:52:37 [error] 681#681: 1252345 peer closed connection in SSL handshake while SSL handshaking to upstream, client: 172.21.0.1, server: symcon.domain.de, request: "POST /hook/ipsviewconnect/api/ HTTP/1.1", upstream: "https://192.168.26.5:3778/hook/ipsviewconnect/api/", host: "symcon.domain.de" 2023/02/12 21:52:37 [error] 681#681: 1252347 peer closed connection in SSL handshake while SSL handshaking to upstream, client: 172.21.0.1, server: symcon.domain.de, request: "POST /hook/ipsviewconnect/api/ HTTP/1.1", upstream: "https://192.168.26.5:3778/hook/ipsviewconnect/api/", host: "symcon.domain.de" 2023/02/12 21:52:38 [error] 681#681: 1252349 peer closed connection in SSL handshake while SSL handshaking to upstream, client: 172.21.0.1, server: symcon.domain.de, request: "POST /hook/ipsviewconnect/api/ HTTP/1.1", upstream: "https://192.168.26.5:3778/hook/ipsviewconnect/api/", host: "symcon.domain.de" 2023/02/12 21:52:38 [error] 681#681: 1252351 peer closed connection in SSL handshake while SSL handshaking to upstream, client: 172.21.0.1, server: symcon.domain.de, request: "POST /hook/ipsviewconnect/api/ HTTP/1.1", upstream: "https://192.168.26.5:3778/hook/ipsviewconnect/api/", host: "symcon.domain.de" 2023/02/12 21:52:38 [error] 681#681: 1252353 peer closed connection in SSL handshake while SSL handshaking to upstream, client: 172.21.0.1, server: symcon.domain.de, request: "POST /hook/ipsviewconnect/api/ HTTP/1.1", upstream: "https://192.168.26.5:3778/hook/ipsviewconnect/api/", host: "symcon.domain.de" 2023/02/12 21:52:38 [error] 681#681: 1252355 peer closed connection in SSL handshake while SSL handshaking to upstream, client: 172.21.0.1, server: symcon.domain.de, request: "POST /hook/ipsviewconnect/api/ HTTP/1.1", upstream: "https://192.168.26.5:3778/hook/ipsviewconnect/api/", host: "symcon.domain.de" 2023/02/12 21:52:39 [error] 681#681: 1252357 peer closed connection in SSL handshake while SSL handshaking to upstream, client: 172.21.0.1, server: symcon.domain.de, request: "POST /hook/ipsviewconnect/api/ HTTP/1.1", upstream: "https://192.168.26.5:3778/hook/ipsviewconnect/api/", host: "symcon.domain.de" 2023/02/12 21:52:39 [error] 681#681: 1252359 peer closed connection in SSL handshake while SSL handshaking to upstream, client: 172.21.0.1, server: symcon.domain.de, request: "POST /hook/ipsviewconnect/api/ HTTP/1.1", upstream: "https://192.168.26.5:3778/hook/ipsviewconnect/api/", host: "symcon.domain.de" 2023/02/12 21:52:39 [error] 682#682: 1252367 peer closed connection in SSL handshake while SSL handshaking to upstream, client: 172.21.0.1, server: symcon.domain.de, request: "POST /hook/ipsviewconnect/api/ HTTP/1.1", upstream: "https://192.168.26.5:3778/hook/ipsviewconnect/api/", host: "symcon.domain.de" 2023/02/12 21:52:39 [error] 681#681: *1252361 peer closed connection in SSL handshake while SSL handshaking to upstream, client: 172.21.0.1, server: symcon.domain.de, request: "POST /hook/ipsviewconnect/api/ HTTP/1.1", upstream: "https://192.168.26.5:3778/hook/ipsviewconnect/api/", host: "symcon.domain.de"

ValentinAUCLERC commented 1 year ago

Hello, I also have the random problem with the 502 errors. I have that since the last update on all proxies. I can just press F5 a few times or wait. That's pretty annoying :-(

Before the update everything was working fine!!

Errlog:

2023/02/12 21:52:37 [error] 681#681: *1252331 upstream prematurely closed connection while reading response header from upstream, client: 172.21.0.1, server: symcon.domain.de, request: "POST /hook/ipsviewconnect/api/ HTTP/1.1", upstream: "https://192.168.26.5:3778/hook/ipsviewconnect/api/", host: "symcon.domain.de" 2023/02/12 21:52:37 [error] 682#682: "

I had this problem too (and mixed hostname resolution... when i was trying to access host1 it was showing host2...) but disabling and enabling again the host was a workaround for me

frap129 commented 1 year ago

Having the same issue. NPM worked fine when I was using docker, but ever since I switched to podman for better rootless containers, I keep getting random 502s that go away after a refresh or two. Unlike the others that have this issue, I'm not getting any error logs

bvn13 commented 1 year ago

Hi! I have the same problem on fresh install using this compose file:

version: '3.7'

networks:
  nginx-proxy-manager:
    external: true

services:
  npm:
    image: 'jc21/nginx-proxy-manager:latest'
    container_name: nginx-proxy-manager
    restart: unless-stopped
    ports:
      - '80:80'
      - '43013:81'
      - '443:443'
    networks:
      - nginx-proxy-manager
    depends_on:
      - npm-db
    environment:
      DB_MYSQL_HOST: npm-db
      DB_MYSQL_PORT: 3306
      DB_MYSQL_USER: npm
      DB_MYSQL_PASSWORD: npm
      DB_MYSQL_NAME: npm
      # Uncomment this if IPv6 is not enabled on your host
      #DISABLE_IPV6: 'true'
    volumes:
      - ./data:/data
      - ./letsencrypt:/etc/letsencrypt

  npm-db:
    image: 'jc21/mariadb-aria:latest'
    restart: unless-stopped
    environment:
      MYSQL_ROOT_PASSWORD: npm
      MYSQL_DATABASE: npm
      MYSQL_USER: npm
      MYSQL_PASSWORD: npm
    networks:
      - nginx-proxy-manager
    volumes:
      - ./data/mysql:/var/lib/mysql

Any ideas?

mariadb logs:

MySQL init process done. Ready for start up.

exec /usr/bin/mysqld --user=mysql --console --skip-name-resolve --skip-networking=0
2023-03-29 17:25:33 0 [Note] /usr/bin/mysqld (mysqld 10.4.15-MariaDB) starting as process 1 ...
2023-03-29 17:25:33 0 [ERROR] mysqld: File '/var/lib/mysql/aria_log_control' not found (Errcode: 13 "Permission denied")
2023-03-29 17:25:33 0 [ERROR] mysqld: Got error 'Can't open file' when trying to use aria control file '/var/lib/mysql/aria_log_control'
2023-03-29 17:25:33 0 [ERROR] Plugin 'Aria' init function returned error.
2023-03-29 17:25:33 0 [ERROR] Plugin 'Aria' registration as a STORAGE ENGINE failed.
2023-03-29 17:25:33 0 [Note] Plugin 'InnoDB' is disabled.
2023-03-29 17:25:33 0 [Note] Plugin 'FEEDBACK' is disabled.
2023-03-29 17:25:33 0 [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded
2023-03-29 17:25:33 0 [ERROR] Failed to initialize plugins.
2023-03-29 17:25:33 0 [ERROR] Aborting
[i] pre-init.d - processing /scripts/pre-init.d/01_secret-init.sh
[i] mysqld already present, skipping creation
[i] MySQL directory already present, skipping creation
2023-03-29 17:25:34 0 [Note] /usr/bin/mysqld (mysqld 10.4.15-MariaDB) starting as process 1 ...
2023-03-29 17:25:34 0 [Note] Plugin 'InnoDB' is disabled.
2023-03-29 17:25:34 0 [Note] Plugin 'FEEDBACK' is disabled.
2023-03-29 17:25:34 0 [Note] Server socket created on IP: '::'.
2023-03-29 17:25:34 0 [Warning] 'user' entry '@3726bb8bb89f' ignored in --skip-name-resolve mode.
2023-03-29 17:25:34 0 [Warning] 'proxies_priv' entry '@% root@3726bb8bb89f' ignored in --skip-name-resolve mode.
2023-03-29 17:25:34 0 [Note] Reading of all Master_info entries succeeded
2023-03-29 17:25:34 0 [Note] Added new Master_info '' to hash table
2023-03-29 17:25:34 0 [Note] /usr/bin/mysqld: ready for connections.
Version: '10.4.15-MariaDB'  socket: '/run/mysqld/mysqld.sock'  port: 3306  MariaDB Server
2023-03-29 17:25:39 3 [Warning] Aborted connection 3 to db: 'unconnected' user: 'unauthenticated' host: '172.19.0.3' (This connection closed normally without authentication)
2023-03-29 17:25:40 4 [Warning] Aborted connection 4 to db: 'unconnected' user: 'unauthenticated' host: '172.19.0.3' (This connection closed normally without authentication)
2023-03-29 17:25:41 5 [Warning] Aborted connection 5 to db: 'unconnected' user: 'unauthenticated' host: '172.19.0.3' (This connection closed normally without authentication)
2023-03-29 17:25:42 6 [Warning] Aborted connection 6 to db: 'unconnected' user: 'unauthenticated' host: '172.19.0.3' (This connection closed normally without authentication)
2023-03-29 17:25:43 7 [Warning] Aborted connection 7 to db: 'unconnected' user: 'unauthenticated' host: '172.19.0.3' (This connection closed normally without authentication)

NPM logs:

❯ Starting nginx ...
❯ Starting backend ...
s6-rc: info: service frontend successfully started
s6-rc: info: service nginx successfully started
s6-rc: info: service backend successfully started
s6-rc: info: service legacy-services: starting
s6-rc: info: service legacy-services successfully started
[3/29/2023] [5:25:33 PM] [Global   ] › ℹ  info      Using MySQL configuration
[3/29/2023] [5:25:33 PM] [Global   ] › ℹ  info      Creating a new JWT key pair...
[3/29/2023] [5:25:38 PM] [Global   ] › ℹ  info      Wrote JWT key pair to config file: /data/keys.json
[3/29/2023] [5:25:39 PM] [Global   ] › ✖  error     Packets out of order. Got: 1 Expected: 0
[3/29/2023] [5:25:40 PM] [Global   ] › ✖  error     Packets out of order. Got: 1 Expected: 0
[3/29/2023] [5:25:41 PM] [Global   ] › ✖  error     Packets out of order. Got: 1 Expected: 0
[3/29/2023] [5:25:42 PM] [Global   ] › ✖  error     Packets out of order. Got: 1 Expected: 0
[3/29/2023] [5:25:43 PM] [Global   ] › ✖  error     Packets out of order. Got: 1 Expected: 0
[3/29/2023] [5:25:44 PM] [Global   ] › ✖  error     Packets out of order. Got: 1 Expected: 0
[3/29/2023] [5:25:45 PM] [Global   ] › ✖  error     Packets out of order. Got: 1 Expected: 0
[3/29/2023] [5:25:46 PM] [Global   ] › ✖  error     Packets out of order. Got: 1 Expected: 0
Javierkaiser commented 1 year ago

Same problem, using version 2.9.22 with Podman. Disabling and re-enabling did not work.

bvn13 commented 1 year ago

for fresh installation prefer to use 2.9.22 version. it works.

matheusware commented 1 year ago

Just to add to the discussion, I also experienced this on the latest version but managed to work around it by creating a shared bridge network on podman(so I can set ip ranges and stuff), attaching the containers to it and then setting static IPs on the containers. With that I can point nginx-proxy-manager to the static ip of the container instead of the hostname and it seems to work reliably.

edit: FWIW I'm using podman 4.2.0 on Rocky Linux 9.1 and configuring everything with ansible so I'm not sure how applicable this is to everyone else.

frap129 commented 1 year ago

Just to add to the discussion, I also experienced this on the latest version but managed to work around it by creating a shared bridge network on podman(so I can set ip ranges and stuff), attaching the containers to it and then setting static IPs on the containers. With that I can point nginx-proxy-manager to the static ip of the container instead of the hostname and it seems to work reliably.

edit: FWIW I'm using podman 4.2.0 on Rocky Linux 9.1 and configuring everything with ansible so I'm not sure how applicable this is to everyone else.

Interesting, I'm also using podman. Are you running rootless? I'm wondering if it's some weirdness related to rootless networking

ootrey commented 1 year ago

Just to add to the discussion, I also experienced this on the latest version but managed to work around it by creating a shared bridge network on podman(so I can set ip ranges and stuff), attaching the containers to it and then setting static IPs on the containers. With that I can point nginx-proxy-manager to the static ip of the container instead of the hostname and it seems to work reliably. edit: FWIW I'm using podman 4.2.0 on Rocky Linux 9.1 and configuring everything with ansible so I'm not sure how applicable this is to everyone else.

Interesting, I'm also using podman. Are you running rootless? I'm wondering if it's some weirdness related to rootless networking

This doesn't seem to be a issue of rootless network. I'm also using Podman but in root permission. NPM works perfectly when using internal IP address instead of hostnames. The problem of occasional 502 errors occur only for containers using hostnames.

calcium90 commented 1 year ago

I'm also experiencing this issue.

docker.io/jc21/nginx-proxy-manager:2.9.22 podman version 4.4.4 OpenSUSE Leap 15.4 kernel 5.14.21-150400.24.60-default cni-plugin-dnsname-1.3.1 RPM installed to enable podman DNS functionality

Created a bridge network with default settings and attached containers for nginx-proxy-manager, grafana and prometheus.

I'd say approx 10-20% of requests fail with a 502, and inspecting the nginx-proxy-manager proxy host error logs shows:

2023/04/24 14:10:46 [error] 713#713: *324 grafana.dns.podman could not be resolved (3: Host not found), client: 172.26.0.131, server: monitoring.redacted.com, request: "GET /api/live/ws HTTP/1.1", host: "monitoring.redacted.com"

However from within the nginx-proxy-manager container itself, nslookup can resolve it every time

$ nslookup grafana.dns.podman
Server:     10.89.0.1
Address:    10.89.0.1:53

Name:   grafana.dns.podman
Address: 10.89.0.2

and this call to grafana's health endpoint succeeds 100% of the time:

$ wget -O - grafana.dns.podman:3000/api/health
Connecting to grafana.dns.podman:3000 (10.89.0.2:3000)
writing to stdout
{
  "commit": "4add91f03d",
  "database": "ok",
  "version": "9.4.7"
-                    100% |***********************************************************************************************************************************************************************************************|    70  0:00:00 ETA
written to stdout

I have a second proxy host configured to a prometheus container, and this also suffers the same problem. I've also tried running all containers as privileged but the same issue occurs. An obvious workaround is to use static container IPs and set these as the forward ip instead of the hostname but it's not ideal.

cpuks commented 1 year ago

Same problem here with fresh 2.10.2 docker install. The problem is solved as soon as I add root domain to one of proxy hosts - as soon as I remove root domain and leave subdomains only I get 502 error image

So basically I cannot use NPM with only subdomains

jbmorgado commented 1 year ago

I'm also experiencing the issues described when using Podman (on plain Docker it worked fine).

Changing the reverse proxy to Caddy instead of Nginx Proxy Manager completely solved the issues, so my guess is that it is due to the DNS resolver NPM is using.

noelmiller commented 1 year ago

Just to add to the discussion, I also experienced this on the latest version but managed to work around it by creating a shared bridge network on podman(so I can set ip ranges and stuff), attaching the containers to it and then setting static IPs on the containers. With that I can point nginx-proxy-manager to the static ip of the container instead of the hostname and it seems to work reliably. edit: FWIW I'm using podman 4.2.0 on Rocky Linux 9.1 and configuring everything with ansible so I'm not sure how applicable this is to everyone else.

Interesting, I'm also using podman. Are you running rootless? I'm wondering if it's some weirdness related to rootless networking

This doesn't seem to be a issue of rootless network. I'm also using Podman but in root permission. NPM works perfectly when using internal IP address instead of hostnames. The problem of occasional 502 errors occur only for containers using hostnames.

I am also experiencing issues when using hostnames. I just switched over to Podman, but specifying subuid and subgid and running under root. When switching to using IPs, the problem seems to have gone away.

pablodgonzalez commented 1 year ago

Since I've pulled the last version I begun to get the random 502. I don;t known exactly why, but the issue is with the domain name resolution.

fuzzyfox commented 1 year ago

Pretty sure this issue and the hosts issue (#2197 ) are related, I've added an interim solution to the other ticket (podman specific) I've not experienced this under docker myself.

MrMasrozYTLIVE commented 9 months ago

Have Portainer Business Edition 2.19.4 and jc21/nginx-proxy-manager:2.10.4. Same issue

U.P.D. method provided by @fuzzyfox actually seems to work.

RobertoMaurizzi commented 7 months ago

Pretty sure this issue and the hosts issue (#2197 ) are related, I've added an interim solution to the other ticket (podman specific) I've not experienced this under docker myself.

I just tried a version of this solution with Docker: starting with a default Docker configuration (i.e. no /etc/docker/daemon.json file) at some version upgrade in the past few months I started getting a lot of 502 caused by host not found errors. Thinking that Docker and resolved hosed something I created the daemon.json file with 2 DNS entries... but that still failed to resolve hostnames very often (half the times?). Taking a hint from #2197 I changed the file to contain only my local router's DNS... and that fixed the problem.

DragonPi commented 3 months ago

I encounter the random 502 errors also.

No Podman, just plain docker. However it seems they only show up when setting: network_mode: host in my docker-compose.yml

Using the 'latest' image

EDIT: Forgot to mention that the 502 is what I see in the different (firefox, chrome, ... also both in-private and not) browsers. In the logs I find 404 host can not be resolved. My first idea was that there was some 'DNS domain clash' (if that even exists). As publicly served hosts as blah.domain.com resolve internally to a local IP (192.168.xxx.xxx range) So I redeployed internal DNS so that the hosts resolve to another domain to rule that out. => But no success