Reverse proxy to another container 502 Bad Gateway

[jpmeijers]

Expected Behavior

When I configure Nginx to reverse proxy to another container using the other container's name, I expect the reverse proxy to always work. Even when the other container restarts.

Current Behavior

If the other container restarts after nginx/letsencrypt started up, the other container can't be found and I get the following error when trying to access it via nginx:

502 Bad Gateway
nginx/1.16.1

Steps to Reproduce

1. docker-compose with three services. RabbitMQ, which take a while before it accepts connections. Other service that depends on Rabbit, and will restart if it can't connect. Letsencrypt that should reverse proxy the other service.
2. docker-compose up -d
3. Try and access the other service via nginx.

(files attached)

4. docker-compose rm -f -s -v letsencrypt
5. docker-compose up -d
6. Now I can access the other service via nginx

Environment

OS:
CPU architecture: x86_64/arm32/arm64
How docker service was installed:

Docker on Ubuntu from apt repo.

Command used to create docker container (run/create/compose/screenshot)

docker-compose up -d

Docker logs

docker-compose.txt docker_logs_letsencrypt.txt nginx_site-confs_default.txt

[dchevell]

I’m wondering if this has to do with the way Docker containers resolve DNS. In the “normal” world, if a host is down their DNS name will still resolve. Restart a server and ping it, and you’ll notice ping can still find its IP address, and just returns “no route to host” or something along those lines. But a container is different - when it’s up, it can be reached via the container name, however when it’s down it basically ceases to exist.

With that being the case … what happens if you configure nginx inside the container to rely on Docker’s own internal DNS (127.0.0.11)? e.g. here’s my nginx.conf:

$ docker exec nginx cat /config/nginx/nginx.conf
## Version 2019/12/19 - Changelog: https://github.com/linuxserver/docker-letsencrypt/commits/master/root/defaults/nginx.conf

user abc;
worker_processes 4;
pid /run/nginx.pid;
include /etc/nginx/modules/*.conf;

events {
        worker_connections 768;
        # multi_accept on;
}

http {
        ##### Use Docker daemon’s DNS resolver
        resolver 127.0.0.11;

(I’m not including the complete file here, obviously) That last line, resolver 127.0.0.11 is the only change I’ve made. If you do this, do you get a different result?

Note: I’m not a contributor to this project, just another user.

[jpmeijers]

Thanks for the suggestion. I know very little about these intrecate inner workings of docker.

I have tried your suggestion to add resolver 127.0.0.11; into the http section of nginx.conf. After that I did a docker-compose down and docker-compose up -d. Hitting the endpoint I get the same 502 Bad Gateway as before.

Doing a docker-compose rm -f -s -v letsencrypt and docker-compose up -d brings the endpoint back up.

[woltsu]

I have the same issue. I am using watchtower to automatically pull new images from docker hub and in the process watchtower restarts the updated containers. After the containers have restarted, Nginx returns 502 from the upstreams. I tried adding the resolver to the http and location blocks, but no difference. After restarting the letsencrypt container, everything works as before.

[j0nnymoe]

@woltsu 502 means the nginx instance can't find your other container, I suspect watchtower is breaking this when it tries to do automatic updates.

[jpmeijers]

I'm updating my containers with docker-compose up -d --build when I get this error. So both docker-compose and watchtower breaks this? I would think the issue is on the letsencrypt container's side, as my other containers don't suffer from this same issue, not being able to find other containers. A case in point is a rabbitmq container. At startup it will restart a couple of times, but my other containers can always find it even after it restarted.

[woltsu]

@j0nnymoe I can replicate this by just doing docker-compose restart to the non-letsencrypt containers. The same issue will arise.

[woltsu]

And what's interesting is that this sometimes happens and other times not. This makes the debugging especially cumbersome, because when I update the Nginx config and it seems to work properly at the time, then the next time it once again returns 502 after the containers restart. This currently breaks our CI/CD automatic deployment, because we need to restart Nginx manually.

[woltsu]

Here I ping the container from the letsencrypt container before the restart:

And here I ping the container after the container has restarted

But nginx still returns 502 (until I restart it)

Here are some configurations:

upstream container-upstream {
        server container:5000 max_fails=18 fail_timeout=10s;
}

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    root /config/www;
    index index.html index.htm index.php;

    server_name x.y.z;

    # enable subfolder method reverse proxy confs
    include /config/nginx/proxy-confs/*.subfolder.conf;

    # all ssl related config moved to ssl.conf
    include /config/nginx/ssl.conf;

    # enable for ldap auth
    #include /config/nginx/ldap.conf;

    client_max_body_size 0;

    location / {
                resolver 127.0.0.11 valid=30s ipv6=off; # I have also added this to the http block, no difference
        limit_req zone=reqlimit burst=20 nodelay;
                proxy_pass http://container-upstream;
                proxy_set_header    Host                $http_host;
                proxy_set_header    X-Real-IP           $remote_addr;
                proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;
    }
}

[LBegnaud]

this isn't an issue with this container/nginx. Just a configuration problem on your end. according to this answer you can put your proxy_pass as a variable, and this will allow it to respect your resolver configuration.

[jpmeijers]

That's what I tried to do in the config I attached to the first message in the issue.

upstream ttnmapper {
    server ingress-api:8080;
}

    # Reverse proxy the root of the subdomain to the ingress api
    location / {
        proxy_pass  http://ttnmapper/v1/;
        proxy_set_header    Host                $http_host;
        proxy_set_header    X-Real-IP           $remote_addr;
        proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;
    }

But now that I look at it I wonder if the line proxy_pass http://ttnmapper/v1/; shouldn't be proxy_pass http://$ttnmapper/v1/;.

[jpmeijers]

Looking at the link you shared it seems like Nginx will cache dns results. If a container started, Nginx looked up the ip and cached it, it will give you a long period of it trying to use the wrong IP. So I assume one will have to add the resolver line with a short timeout like 30 seconds.

[aptalca]

@jpmeijers you'd be better off taking one of our preset proxy confs and modifying the app name, port and proto as needed: https://github.com/linuxserver/reverse-proxy-confs/blob/master/adguard.subdomain.conf.sample

[jpmeijers]

The examples want to force me to either use a subdomain or a subfolder, which I neither want. I just want to proxy the root to another container. Basically using the letsencrypt container to server an http endpoint on https.

I changed my config to the following, adding a line for the resolver. This seems to work.

  # Reverse proxy the root of the subdomain to the ingress api
  location / {
      resolver 127.0.0.11 valid=30s;
      proxy_pass  http://ttnmapper/v1/;
      proxy_set_header    Host                $http_host;
      proxy_set_header    X-Real-IP           $remote_addr;
      proxy_set_header    X-Forwarded-For     $proxy_add_x_forwarded_for;
  }

[aptalca]

You can modify it however you like. Also fyi, the heimdall and organizr subfolder confs actually let you reverse proxy at the homepage.

What I'm trying to tell you is that the way we set up the proxy_pass line in combination with the resolver line lets nginx deal with changing IPs, but you do you

[jpmeijers]

Wow, no need to be rude. I'm just asking for help here.

I've followed the readme at https://github.com/linuxserver/docker-letsencrypt#site-config-and-reverse-proxy which states:

The default site config resides at /config/nginx/site-confs/default. Feel free to modify this file, and you can add other conf files to this directory. However, if you delete the default file, a new default will be created on container start.

So that is what I did, seeing as I am not trying to proxy a popular app. After a lot of trial and error I finally had it working, except for this issue where I get the 502 error in some circumstances. I'm still not sure if my last change is a temporary or permanent fix, so I'm not sure if closing this issue is valid. Also @woltsu didn't confirm if the issue is fixed for him or not.

It would be nice if you could document how one should use this letsencrypt container for this simple use case of just wrapping an http endpoint in https with a letsencrypt certificate.

[aptalca]

Rude? No. Mere indifference? Yeah. You refuse to take our advice. So my last suggestion is for you to do whatever suits you.

There is plenty of documentation in the letsencrypt and reverse-proxy-confs repos.

simple use case of just wrapping an http endpoint in https with a letsencrypt certificate that's not really the issue here. The issue is that your http endpoint keeps changing.

I provided the fix for both of you. You (the op) claimed you fixed your issue, hence, issue closed.

[woltsu]

@jpmeijers you'd be better off taking one of our preset proxy confs and modifying the app name, port and proto as needed: https://github.com/linuxserver/reverse-proxy-confs/blob/master/adguard.subdomain.conf.sample

This seems to work (at least when setting the valid parameter to the resolver), but the difference here is that we proxy directly to the container, not to the defined upstream. But this is a better situation than before, so thanks! :)