opam-health-check DNS issue

mtelvers commented 1 month ago

When opam-health-check is deployed or updated the IP address of the containers needs to be manually updated within the Caddy container. I have been finding the address manually with docker inspect

docker inspect infra_opam-health-check.1.7cxsyuvefr48cg2vezs0p5riw --format='{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}’

... then editing the /etc/hosts file in the Caddy container.

The behaviour can be reproduced on the target machine using a docker stack like this:

version: "3.7"
services:
  web1:
    image: nginx
  web2:
    image: nginx

After deploying the stack with docker stack deploy --compose-file stack.yml test, the issue can be seen by attempting to curl between the containers both by name and IP address:

# docker exec test_web1.1.4kph5ho08uaqvqf3s0xppnvce curl -s web2:80
curl: (7) Failed to connect to web2 port 80 after 3075 ms: Couldn't connect to server

# docker exec test_web1.1.4kph5ho08uaqvqf3s0xppnvce curl -s 10.0.2.3:80
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

I cannot reproduce this behaviour on another machine.

This Stack Overflow post explains how the Docker DNS should work with IP Tables and namespaces.

mtelvers commented 1 month ago

Running the Docker daemon in debug mode logs all DNS requests. It was clear that the names were being resolved but to the wrong addresses.

This thread describes the same issue, https://github.com/moby/moby/issues/41766, and while it doesn’t have an explanation, the workaround of adding endpoint_mode: dnsrr works on this machine.

version: "3.7"
services:
  web1:
    image: nginx
    deploy:
      endpoint_mode: dnsrr
  web2:
    image: nginx
    deploy:
      endpoint_mode: dnsrr

mtelvers commented 1 month ago

@neomterry Do try adding deploy: endpoint_mode: dnsrr this worked immediately in my use case.

ocaml / infrastructure

opam-health-check DNS issue #119