dns is not updated when addnhosts change

jedi7 commented 3 years ago

Hi, I'm facing following issue:

Versions:

podman: 3.4.0
dnsname: 1.3.1-1

I have multiple containers connected to network named "podmannet". They have proper and updated ip/name in addnhosts file. But the dnsmasq does not reread this file when changes.

steps to reproduce:

start 2 new containers A, B
ping A from B (successfull)
restart container A, which will get new ip
ping A from B (fails, it still points to old ip)

Possible workaround (untested): use --hostsdir= for dnsmasq which is automatically reread when changes So the "addnhosts" should be in a subdirectory. And use --hostsdir= instead of --addnhosts=

vinzent commented 2 years ago

I can't confirm this behaviour.

started 2 containers. the addnhosts file looks this:

$ cat /run/user/1000/rootless-cni/run/user/1000/containers/cni/dnsname/test/addnhosts
10.89.0.2   container-a
10.89.0.3   container-b

then ran podman stop container-a + podman start container-a.

after that, the addnhsots file content:

$ cat /run/user/1000/rootless-cni/run/user/1000/containers/cni/dnsname/test/addnhosts
10.89.0.3   container-b
10.89.0.4   container-a

container-a has new .4 ip.

looking up the ip of container-a in container-b:

$ podman exec container-b getent hosts container-a
10.89.0.4       container-a.dns.podman

so DNS response contains the new ip correctly.

Versions:

$ rpm -qa podman*
podman-gvproxy-3.4.2-1.fc35.x86_64
podman-plugins-3.4.2-1.fc35.x86_64
podman-3.4.2-1.fc35.x86_64

rhatdan commented 2 years ago

That mean we can close this issue?

jedi7 commented 2 years ago

Hi, please try to recreate the A container.

The real containers are (on arm64)

docker.io/graphiteapp/graphite-statsd:1.1.8-2
docker.io/grafana/grafana:8.2.2

when I recreate the graphite-statsd, then the grafana is not able to connect to the new one (stil resolving the old ip of graphite)

vinzent commented 2 years ago

also if I re-create the container, DNS resolution works fine.

$ podman run --network test -d --name container-a registry.access.redhat.com/ubi8/ubi /bin/sleep 3600
f10321a6664651cf321768f0ab1c38bcb508df5fc8298bc42d433083ac48462a

$ podman run --network test -d --name container-b registry.access.redhat.com/ubi8/ubi /bin/sleep 3600
09693c5898728a4e0ccb20729eaa774362adf01bd3da3be8d38ac33dadb0b427

$ cat /run/user/1000/rootless-cni/run/user/1000/containers/cni/dnsname/test/addnhosts
10.89.0.2   container-a
10.89.0.3   container-b

$ podman stop container-a
container-a

$ podman rm container-a
f10321a6664651cf321768f0ab1c38bcb508df5fc8298bc42d433083ac48462a

$ podman run --network test -d --name container-a registry.access.redhat.com/ubi8/ubi /bin/sleep 3600
c0aa2a8f11b1afd00d2d675a1cd843b3eb6c453b018af26f8656e336affd8cd5

$ cat /run/user/1000/rootless-cni/run/user/1000/containers/cni/dnsname/test/addnhosts
10.89.0.3   container-b
10.89.0.4   container-a

$ podman run --network test -d --name container-a registry.access.redhat.com/ubi8/ubi /bin/sleep 3600

$ podman exec container-b getent hosts container-a
10.89.0.4       container-a.dns.podman

I rather suspect your app doesn't lookup the DNS name again and just tries to connect to the old ip that it looked up earlier?

jedi7 commented 2 years ago

ok that is strange. With your example it is working as expected (I tried also ubuntu:20.04)

but please see this:

[root@srv graphite]# cat /run/user/0/containers/cni/dnsname/podmannet/addnhosts
192.168.0.2     nextcloud
192.168.0.5     notes
192.168.0.4     kanboard
192.168.0.7     gitea
192.168.0.10    grafana
192.168.0.15    container-b
192.168.0.17    graphite
192.168.0.19    container-a
[root@srv graphite]# podman exec -it graphite sh
/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0@if32: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
    link/ether a2:3b:e9:3c:d8:f2 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.18/24 brd 192.168.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::a03b:e9ff:fe3c:d8f2/64 scope link
       valid_lft forever preferred_lft forever
/ #

As you can see, the addnhosts shows now 192.168.0.17 graphite. But when I run shell inside and show ipt, it shows 192.168.0.18

I'm not sure how to debug this :(

when I run podman inspect graphite:

       "Networks": {
            "podmannet": {
                "EndpointID": "",
                "Gateway": "192.168.0.1",
                "IPAddress": "192.168.0.18",
                "IPPrefixLen": 24,
                "IPv6Gateway": "",
                "GlobalIPv6Address": "",
                "GlobalIPv6PrefixLen": 0,
                "MacAddress": "a2:3b:e9:3c:d8:f2",
                "NetworkID": "podmannet",
                "DriverOpts": null,
                "IPAMConfig": null,
                "Links": null
            }
        }

jedi7 commented 2 years ago

Also I somehow get this state, where is the graphite twice in dns. But only one instance.

root@srv ~]# podman ps
CONTAINER ID  IMAGE                                           COMMAND               CREATED             STATUS                 PORTS                                                   NAMES
f245975b72f5  localhost/appdaemon-dev:4.1.0                                         7 weeks ago         Up 15 hours ago                                                                appdaemon
60ea19d90598  localhost/mynextcloud-smb-fpm:21.0.5            /usr/bin/supervis...  7 weeks ago         Up 18 hours ago        0.0.0.0:3036->9000/tcp                                  nextcloud
a830de5e3377  docker.io/grafana/grafana:8.2.2                                       6 weeks ago         Up 3 minutes ago       0.0.0.0:3000->3000/tcp                                  grafana
be964bfae89c  localhost/mykanboard:v1.2.20-2-gb8b324e1-dirty                        6 weeks ago         Up 18 hours ago        0.0.0.0:3034->80/tcp                                    kanboard
b3f7a7043ce6  docker.io/zadam/trilium:0.48.7                  node ./src/www        2 weeks ago         Up 18 hours ago        0.0.0.0:3035->3035/tcp                                  notes
cea2325c3bfd  docker.io/gitea/gitea:1.15.6                    /bin/s6-svscan /e...  18 hours ago        Up 18 hours ago        0.0.0.0:2222->22/tcp, 0.0.0.0:3037->3000/tcp            gitea
93fe73ca6c79  docker.io/graphiteapp/graphite-statsd:1.1.8-2                         About a minute ago  Up About a minute ago  0.0.0.0:3031->80/tcp, 0.0.0.0:2003-2004->2003-2004/tcp  graphite
[root@srv ~]# cat /run/user/0/containers/cni/dnsname/podmannet/addnhosts
192.168.0.2     nextcloud
192.168.0.5     notes
192.168.0.4     kanboard
192.168.0.7     gitea
192.168.0.10    grafana
192.168.0.15    container-b
192.168.0.17    graphite
192.168.0.19    container-a
192.168.0.22    graphite
[root@srv ~]#

And when I reload the network: "podman network reload -a" Then all is fixed and working.

agdevsu commented 2 years ago

Same issue

containers / dnsname

dns is not updated when addnhosts change #85