search

lucaslorentz / caddy-docker-proxy

Caddy as a reverse proxy for Docker
MIT License
2.61k stars 163 forks source link

{{upstreams}} resolving also "localhost" resulting in connection refused issues #592

Closed maxiride closed 3 months ago

maxiride commented 3 months ago

Given the labels of a container that runs a process listening on port 3000 (port inside the container). The application container and the caddy container are in the same custom bridge network.

caddy: example.com
caddy.reverse_proxy: {{upstreams 3000}}

the compiled caddyfile is:

example.com {
    reverse_proxy 10.20.14.2:3000 :3000
}

which results in a 50/50 change of being proxied to the actual container (10.20.14.2:3000) or getting a connection refused when connecting to :3000 which is the caddy container itself. The internal IP of the application container 10.20.14.2 is resolved correctly.

I would expect the caddyfile to be:

example.com {
    reverse_proxy 10.20.14.2:3000
}

Am I configuring wrongly the labels?

Docker version: lucaslorentz/caddy-docker-proxy:2.8.10

I've tried to spin up a whoami container and the compiled caddyfile doesn't show this issue. So I suppose something is wrong with my app container configuration rather than caddy-docker-proxy. Here is the stripped down docker configuration:

{
    "Config": {
        "ExposedPorts": {
            "3000/tcp": {},
            "3001/tcp": {},
            "3002/tcp": {}
        },
        "Hostname": "819fe1365ac8",
        "Labels": {
            "caddy": "example.com",
            "caddy.handle_0": "/api*",
            "caddy.handle_0.reverse_proxy": "{{upstreams 3001}}",
            "caddy.handle_0.uri": "strip_prefix /api",
            "caddy.handle_1": "/approvazione/*",
            "caddy.handle_1.reverse_proxy": "{{upstreams 3002}}",
            "caddy.reverse_proxy": "{{upstreams 3000}}",
            "com.docker.compose.config-hash": "9739846c49df04ee3f106eead75775e727ecd1c52fce2e85ef4bb60590e2dc41",
            "com.docker.compose.container-number": "1",
            "com.docker.compose.oneoff": "False",
            "com.docker.compose.project": "example-com-prod",
            "com.docker.compose.project.config_files": "docker-compose.yml",
            "com.docker.compose.project.working_dir": "/home/ubuntu/compose-files/example-com-prod",
            "com.docker.compose.service": "api",
            "com.docker.compose.version": "1.25.1",
            "org.opencontainers.image.ref.name": "ubuntu",
            "org.opencontainers.image.version": "20.04"
        },
        "OnBuild": null,
        "OpenStdin": false,
        "StdinOnce": false,
        "Tty": false,
        "User": "",
        "Volumes": {
            "/app/backend/o_ldv": {}
        },
        "WorkingDir": ""
    },
    "Name": "/example-com-prod_api",
    "NetworkSettings": {
        "Bridge": "",
        "EndpointID": "",
        "Gateway": "",
        "GlobalIPv6Address": "",
        "GlobalIPv6PrefixLen": 0,
        "HairpinMode": false,
        "IPAddress": "",
        "IPPrefixLen": 0,
        "IPv6Gateway": "",
        "LinkLocalIPv6Address": "",
        "LinkLocalIPv6PrefixLen": 0,
        "MacAddress": "",
        "Networks": {
            "caddy": {
                "Aliases": [
                    "api",
                    "2422bb545350"
                ],
                "DriverOpts": null,
                "EndpointID": "3621a11dcead2fcaaa0828576206c6d4ad8e19ab6109550082896f4d920fb2f8",
                "Gateway": "10.20.14.1",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "IPAMConfig": {},
                "IPAddress": "10.20.14.2",
                "IPPrefixLen": 24,
                "IPv6Gateway": "",
                "Links": null,
                "MacAddress": "02:42:0a:14:0e:02",
                "NetworkID": "939e76b42e2316cf12f71ceef1719b3396c81daeb73526a45825e3e0ab16fd35"
            },
            "example-com-prod_net": {
                "Aliases": [
                    "api",
                    "2422bb545350"
                ],
                "DriverOpts": null,
                "EndpointID": "cc89745965fbfda17a6c05f1c0efd59c5af7ec1ff97324ba8f0224733c5caede",
                "Gateway": "10.20.22.1",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "IPAMConfig": null,
                "IPAddress": "10.20.22.4",
                "IPPrefixLen": 24,
                "IPv6Gateway": "",
                "Links": null,
                "MacAddress": "02:42:0a:14:16:04",
                "NetworkID": "9f0d29f57aa7ea3f473fe40fad8d47ef00e42b1629a58a08f60b86db93225535"
            }
        },
        "Ports": {
            "3000/tcp": [
                {
                    "HostIp": "0.0.0.0",
                    "HostPort": "32828"
                }
            ],
            "3001/tcp": [
                {
                    "HostIp": "0.0.0.0",
                    "HostPort": "32827"
                }
            ],
            "3002/tcp": [
                {
                    "HostIp": "0.0.0.0",
                    "HostPort": "40561"
                }
            ]
        },
        "SandboxID": "43ea047212f5ff3ddcb984dc9d3af24be69b5989e2f28d55be89b14635521934",
        "SandboxKey": "/var/run/docker/netns/43ea047212f5",
        "SecondaryIPAddresses": null,
        "SecondaryIPv6Addresses": null
    }
}

And the compiled caddyfile is below, everything is as expected except for the double entries in the reverse_proxy directives:

example.com {
    handle /api* {
        reverse_proxy 10.20.14.2:3001 :3001
        uri strip_prefix /api
    }
    handle /approvazione/* {
        reverse_proxy 10.20.14.2:3002 :3002
    }
    reverse_proxy 10.20.14.2:3000 :3000
}
maxiride commented 3 months ago

I've tried to update the application labels removing the upstreams token and setting directly the container name:

caddy: example.com
caddy.reverse_proxy: container_name:3000

and the compiled caddyfile from caddy-docker-proxy still has the double entry.

reverse_proxy container_name:3000 :3000

So it's like if the caddyfile is being assembled with data from other sources. I did configured the ENV variable CADDY_DOCKER_CADDYFILE_PATH but there are completly different hostnames and directive so it's impossible that a merge from the base Caddyfile ends up there.

External caddyfile being passed:

exampleA.it {
  root * /mnt/16-share/video/progetto_video
  file_server
}

exampleB.it, exampleC.it {
  reverse_proxy 172.21.15.36:9001
}

web.exampleO.it, mail.exampleD.it {
    reverse_proxy 192.168.0.16
}
maxiride commented 3 months ago

So it's like if the caddyfile is being assembled with data from other sources. I did configured the ENV variable

Found the culprit. The application recently had an upgrade, there as a second container with the same labels that ended up being merged. The odd things is that the second container was stopped several weeks ago, caddy-docker-proxy shouldn't read labels of stopped containers.

maxiride commented 3 months ago

Now that I eventually understood the underlying issue I found the PR https://github.com/lucaslorentz/caddy-docker-proxy/pull/560, but it has not been released yet.

Being a pretty breaking change I would have hoped it urged a new release sooner. 😢

lucaslorentz commented 3 months ago

@maxiride You're right, that is indeed a quite relevant fix. Releases are automated and straight forward, I just missed that this fix was in the queue.

Doing a release right now: https://github.com/lucaslorentz/caddy-docker-proxy/releases/tag/v2.8.11