502 Bad Gateway & Upstream connection refused

[ethanbeyer]

Quick Overview

I use Nginx Proxy for local web development. It allows me to create "pretty URLs" for the different apps and websites I'm working on. I've had a running Nginx Proxy for the better part of a year, but recently needed to add a new virtualhost, and since then I cannot get Nginx Proxy to load anything other than 502 Bad Gateway.

As far as I can tell, I have configured these different containers correctly, but I continually see these errors in the nginx-proxy's logs:

2024-01-24 10:25:11 nginx.1     | whoami.local 192.168.65.1 - - [24/Jan/2024:15:25:11 +0000] "GET / HTTP/1.1" 502 157 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:121.0) Gecko/20100101 Firefox/121.0" "172.18.0.3:8400"
2024-01-24 10:25:11 nginx.1     | 2024/01/24 15:25:11 [error] 25#25: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.65.1, server: whoami.local, request: "GET / HTTP/1.1", upstream: "http://172.18.0.3:8400/", host: "whoami.local"

Can someone point out where I've gone wrong? Configs attached.

Configs

Nginx Proxy docker-compose.yml:

version: '3'

services:
  proxy:
    image: nginxproxy/nginx-proxy:1.4
    build:
      context: ./
      dockerfile: Dockerfile
    ports:
      - "80:80"
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro
    networks:
      - nginx-proxy
    restart: always

networks:
  nginx-proxy:
    name: local-dev
    driver: bridge

Test Whoami docker-compose.yml:

version: '3'

services:

  whoami:
    image: jwilder/whoami
    expose:
      - "8400"
    environment:
      - VIRTUAL_HOST=whoami.local
      - VIRTUAL_PORT=8400
    networks:
      - local-dev

networks:
  local-dev:
    external: true

Local hosts file:

127.0.0.1    whoami.local

Generated Nginx Configs from nginx-proxy Container

# nginx-proxy version : 1.4.0-71-gd46881f
# Networks available to the container running docker-gen (which are assumed to
# match the networks available to the container running nginx):
#     local-dev
# If we receive X-Forwarded-Proto, pass it through; otherwise, pass along the
# scheme used to connect to this server
map $http_x_forwarded_proto $proxy_x_forwarded_proto {
    default $http_x_forwarded_proto;
    '' $scheme;
}
map $http_x_forwarded_host $proxy_x_forwarded_host {
    default $http_x_forwarded_host;
    '' $host;
}
# If we receive X-Forwarded-Port, pass it through; otherwise, pass along the
# server port the client connected to
map $http_x_forwarded_port $proxy_x_forwarded_port {
    default $http_x_forwarded_port;
    '' $server_port;
}
# If the request from the downstream client has an "Upgrade:" header (set to any
# non-empty value), pass "Connection: upgrade" to the upstream (backend) server.
# Otherwise, the value for the "Connection" header depends on whether the user
# has enabled keepalive to the upstream server.
map $http_upgrade $proxy_connection {
    default upgrade;
    '' $proxy_connection_noupgrade;
}
map $upstream_keepalive $proxy_connection_noupgrade {
    # Preserve nginx's default behavior (send "Connection: close").
    default close;
    # Use an empty string to cancel nginx's default behavior.
    true '';
}
# Abuse the map directive (see <https://stackoverflow.com/q/14433309>) to ensure
# that $upstream_keepalive is always defined.  This is necessary because:
#   - The $proxy_connection variable is indirectly derived from
#     $upstream_keepalive, so $upstream_keepalive must be defined whenever
#     $proxy_connection is resolved.
#   - The $proxy_connection variable is used in a proxy_set_header directive in
#     the http block, so it is always fully resolved for every request -- even
#     those where proxy_pass is not used (e.g., unknown virtual host).
map "" $upstream_keepalive {
    # The value here should not matter because it should always be overridden in
    # a location block (see the "location" template) for all requests where the
    # value actually matters.
    default false;
}
# Apply fix for very long server names
server_names_hash_bucket_size 128;
# Default dhparam
ssl_dhparam /etc/nginx/dhparam/dhparam.pem;
# Set appropriate X-Forwarded-Ssl header based on $proxy_x_forwarded_proto
map $proxy_x_forwarded_proto $proxy_x_forwarded_ssl {
    default off;
    https on;
}
gzip_types text/plain text/css application/javascript application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
log_format vhost '$host $remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$upstream_addr"';
access_log off;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305';
    ssl_prefer_server_ciphers off;
error_log /dev/stderr;
resolver 127.0.0.11;
# HTTP 1.1 support
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $proxy_connection;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $proxy_x_forwarded_host;
proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
proxy_set_header X-Forwarded-Ssl $proxy_x_forwarded_ssl;
proxy_set_header X-Forwarded-Port $proxy_x_forwarded_port;
proxy_set_header X-Original-URI $request_uri;
# Mitigate httpoxy attack (see README for details)
proxy_set_header Proxy "";
server {
    server_name _; # This is just an invalid value which will never trigger on a real hostname.
    server_tokens off;
    access_log /var/log/nginx/access.log vhost;
    http2 on;
    listen 80;
    listen 443 ssl;
    ssl_session_cache shared:SSL:50m;
    ssl_session_tickets off;
    # No default.crt certificate found for this vhost, so force nginx to emit a
    # TLS error if the client connects via https.
    ssl_ciphers aNULL;
    set $empty "";
    ssl_certificate data:$empty;
    ssl_certificate_key data:$empty;
    if ($https) {
        return 444;
    }
    return 503;
}
# whoami.local/
upstream whoami.local {
    # Container: test-whoami-1
    #     networks:
    #         local-dev (reachable)
    #     IP address: 172.18.0.3
    #     exposed ports: 8000/tcp 8400/tcp
    #     default port: 80
    #     using port: 8400
    server 172.18.0.3:8400;
}
server {
    server_name whoami.local;
    access_log /var/log/nginx/access.log vhost;
    http2 on;
    listen 80 ;
    listen 443 ssl ;
    # No certificate found for this vhost, so force nginx to emit a TLS error if
    # the client connects via https.
    ssl_ciphers aNULL;
    set $empty "";
    ssl_certificate data:$empty;
    ssl_certificate_key data:$empty;
    if ($https) {
        return 444;
    }
    location / {
        proxy_pass http://whoami.local;
        set $upstream_keepalive false;
    }
}

[buchdag]

Hi.

As far as I remember, jwilder/whoami does not listen on port 8400 but 8000, so you're basically telling / forcing the proxy to connect to a port where nothing is listening.

Edit: we made a change last year (I think) that forces the proxy to honor the VIRTUAL_PORT, even when this is a wrong value like here, rather than doing some automagical configuration. It's possible that this test case was working prior to this change because the proxy was not honoring the 8400 VIRTUAL_PORT.

[ethanbeyer]

Thanks for the reply @buchdag! I changed the two 8400 to 8000 in the whoami docker-compose.yml and it loaded. 🎉 So that's a win. Thank you again.

whoami was a lightweight proof-of-concept to debug why a different docker compose project isn't working.

I'm going to post a second comment detailing the configs for that project. It seems like another port/virtual port issue but I'm stuck.

[ethanbeyer]

Proxy Configs (again, for convenience)

version: '3'

services:
  proxy:
    image: nginxproxy/nginx-proxy:1.4
    build:
      context: ./
      dockerfile: Dockerfile
    ports:
      - "80:80"
      # - "443:443"
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro
    networks:
      - nginx-proxy
    restart: always

networks:
  nginx-proxy:
    name: local-dev
    driver: bridge

Separate Docker Compose Project

version: '3'

services:
    http:
        image: php:7.2-apache
        container_name: direxamplelocal_http #@TODO
        build:
            context: ./docker/http
            dockerfile: Dockerfile
        environment:
            VIRTUAL_HOST: dir.example.local
            VIRTUAL_PORT: 8089
        volumes:
            - './:/var/www/html'
        networks:
            - local-dev # nginx-proxy network
            - example-directory
        expose:
            - 8089

    mysql:
        image: mysql:5.7
        container_name: ${DB_HOST}
        platform: linux/x86_64
        ports:
            - ${DB_PORT}:3306
        environment:
            MYSQL_DATABASE: ${DB_NAME}
            MYSQL_USER: ${DB_USER}
            MYSQL_PASSWORD: ${DB_PASSWORD}
            MYSQL_RANDOM_ROOT_PASSWORD: '1'
        volumes:
            - './docker/mysql/init:/docker-entrypoint-initdb.d/' # startup volume
            - 'db_dataexample:/var/lib/mysql' # persistent storage of the DB
        networks:
            - example-directory

volumes:
    db_dataexample:
        driver: local

networks:
    local-dev:
        external: true # nginx-proxy network
    example-directory:
        driver: bridge

Hosts

127.0.0.1    dir.example.local

Nginx Configs generated within nginx-proxy

# nginx-proxy version : 1.4.0-71-gd46881f
# Networks available to the container running docker-gen (which are assumed to
# match the networks available to the container running nginx):
#     local-dev
# If we receive X-Forwarded-Proto, pass it through; otherwise, pass along the
# scheme used to connect to this server
map $http_x_forwarded_proto $proxy_x_forwarded_proto {
    default $http_x_forwarded_proto;
    '' $scheme;
}
map $http_x_forwarded_host $proxy_x_forwarded_host {
    default $http_x_forwarded_host;
    '' $host;
}
# If we receive X-Forwarded-Port, pass it through; otherwise, pass along the
# server port the client connected to
map $http_x_forwarded_port $proxy_x_forwarded_port {
    default $http_x_forwarded_port;
    '' $server_port;
}
# If the request from the downstream client has an "Upgrade:" header (set to any
# non-empty value), pass "Connection: upgrade" to the upstream (backend) server.
# Otherwise, the value for the "Connection" header depends on whether the user
# has enabled keepalive to the upstream server.
map $http_upgrade $proxy_connection {
    default upgrade;
    '' $proxy_connection_noupgrade;
}
map $upstream_keepalive $proxy_connection_noupgrade {
    # Preserve nginx's default behavior (send "Connection: close").
    default close;
    # Use an empty string to cancel nginx's default behavior.
    true '';
}
# Abuse the map directive (see <https://stackoverflow.com/q/14433309>) to ensure
# that $upstream_keepalive is always defined.  This is necessary because:
#   - The $proxy_connection variable is indirectly derived from
#     $upstream_keepalive, so $upstream_keepalive must be defined whenever
#     $proxy_connection is resolved.
#   - The $proxy_connection variable is used in a proxy_set_header directive in
#     the http block, so it is always fully resolved for every request -- even
#     those where proxy_pass is not used (e.g., unknown virtual host).
map "" $upstream_keepalive {
    # The value here should not matter because it should always be overridden in
    # a location block (see the "location" template) for all requests where the
    # value actually matters.
    default false;
}
# Apply fix for very long server names
server_names_hash_bucket_size 128;
# Default dhparam
ssl_dhparam /etc/nginx/dhparam/dhparam.pem;
# Set appropriate X-Forwarded-Ssl header based on $proxy_x_forwarded_proto
map $proxy_x_forwarded_proto $proxy_x_forwarded_ssl {
    default off;
    https on;
}
gzip_types text/plain text/css application/javascript application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
log_format vhost '$host $remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$upstream_addr"';
access_log off;
    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305';
    ssl_prefer_server_ciphers off;
error_log /dev/stderr;
resolver 127.0.0.11;
# HTTP 1.1 support
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $proxy_connection;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $proxy_x_forwarded_host;
proxy_set_header X-Forwarded-Proto $proxy_x_forwarded_proto;
proxy_set_header X-Forwarded-Ssl $proxy_x_forwarded_ssl;
proxy_set_header X-Forwarded-Port $proxy_x_forwarded_port;
proxy_set_header X-Original-URI $request_uri;
# Mitigate httpoxy attack (see README for details)
proxy_set_header Proxy "";
server {
    server_name _; # This is just an invalid value which will never trigger on a real hostname.
    server_tokens off;
    access_log /var/log/nginx/access.log vhost;
    http2 on;
    listen 80;
    listen 443 ssl;
    ssl_session_cache shared:SSL:50m;
    ssl_session_tickets off;
    # No default.crt certificate found for this vhost, so force nginx to emit a
    # TLS error if the client connects via https.
    ssl_ciphers aNULL;
    set $empty "";
    ssl_certificate data:$empty;
    ssl_certificate_key data:$empty;
    if ($https) {
        return 444;
    }
    return 503;
}
# dir.example.local/
upstream dir.example.local {
    # Container: direxamplelocal_http
    #     networks:
    #         direxamplelocal_example-directory (unreachable)
    #         local-dev (reachable)
    #     IP address: 172.18.0.3
    #     exposed ports: 80/tcp 8089/tcp
    #     default port: 80
    #     using port: 8089
    server 172.18.0.3:8089;
}
server {
    server_name dir.example.local;
    access_log /var/log/nginx/access.log vhost;
    http2 on;
    listen 80 ;
    listen 443 ssl ;
    # No certificate found for this vhost, so force nginx to emit a TLS error if
    # the client connects via https.
    ssl_ciphers aNULL;
    set $empty "";
    ssl_certificate data:$empty;
    ssl_certificate_key data:$empty;
    if ($https) {
        return 444;
    }
    location / {
        proxy_pass http://dir.example.local;
        set $upstream_keepalive false;
    }
}

I'm not sure if this is another expose/VIRTUAL_PORT issue or what. I've tried 80, 8000, 8001, 8080, and 8089. Sometimes the browser resolves to the proxy's 502, and other times it says "Unable to connect" with nothing in the proxy's stdout or stderr.

I'm sure it's a misconfiguration issue for me, but I just don't know what to do to fix it. I've tried everything I know.

Edit: fixing a typo in the configs

[ethanbeyer]

...Forgot to add this detail.

No matter what I do, http://dir.example.local always tries to resolve to https://dir.example.local (https). I think that is part of the issue.

[buchdag]

No matter what I do,http://dir.example.local always tries to resolve to https://dir.example.local (https). I think that is part of the issue.

You did not provide any certificate and private key so yes, HTTPS won't work.

Your browser might have cached HSTS for this domain from a previous test, if that's the case it will always try to use HTTPS until HTST expires or you empty your browser HSTS cache..

[kewldan]

Same issue, 502 Gateway Error,

Nginx-proxy run command: docker run --detach --name nginx-proxy --publish 80:80 --volume /var/run/docker.sock:/tmp/docker.sock:ro nginxproxy/nginx-proxy:1.5

Everything works if i run my web server in containers and set env VIRTUAL_HOST to my domain name, but once i have to use docker compose to my web project based on NextJS frontend and FastAPI backend. Before web launch i had 503 code, thats mean that proxy didnt find container with VIRTUAL HOST of domain, thats OK. But after start i got 502 error and found this issue. I wrote docker compose to reproduce this problem:

services:
  web:
    build: .
    restart: unless-stopped
    environment:
      - VIRTUAL_HOST=example.com
      - VIRTUAL_PORT=80
    ports:
      - 8080:80 # Port 80 already in use by nginx-proxy container
    expose:
      - 80:80

When i started compose i run command to test web server curl -H "Host: example.com" localhost but i got:

<html>
<head><title>502 Bad Gateway</title></head>
<body>
<center><h1>502 Bad Gateway</h1></center>
<hr><center>nginx/1.26.0</center>
</body>
</html>

But web server is launched and i can prove it by run command curl localhost:8080 and i got:

{"detail":"Not Found"}

Nginx-proxy logs:

nginx.1     | 2024/06/04 13:05:26 [error] 149#149: *627 no live upstreams while connecting to upstream, client: 172.17.0.1, server: example.com, request: "GET / HTTP/1.1", upstream: "http://example.com/", host: "example.com"
nginx.1     | example.com 172.17.0.1 - - [04/Jun/2024:13:05:26 +0000] "GET / HTTP/1.1" 502 157 "-" "curl/7.81.0" "example.com"

So its not a problem with HSTS or anything

FULL TEST REPO: https://github.com/kewldan/composeTest

[buchdag]

Please verify that your nginx-proxy and web container share at least one Docker network.