search

weaveworks / weave

Simple, resilient multi-host containers networking and more.
https://www.weave.works
Apache License 2.0
6.62k stars 671 forks source link

DNS entries are not removed with version 2.5.x after container has gone. Works fine in version 2.4.1. #3652

Open bernhara opened 5 years ago

bernhara commented 5 years ago

What you expected to happen?

After a container hase gone (has been rm-ed), the DNS should no more show a valid hostname for this container

What happened?

I start the operation right after having started and "reset" weavenet.

I launch 8 containers simultaneously, using the basic "docker run" command.

_CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 15473e656da3 s-eunuc:5000/dip/mlr-worker:x86_64-latest "/w/w /home/dip/bin/…" 5 seconds ago Up 4 seconds mlr_worker_05 cb7132a1be3f s-eunuc:5000/dip/mlr-worker:x86_64-latest "/w/w /home/dip/bin/…" 5 seconds ago Up 4 seconds mlr_worker_06 129240b4b9bc s-eunuc:5000/dip/mlr-worker:x86_64-latest "/w/w /home/dip/bin/…" 5 seconds ago Up 4 seconds mlr_worker_07 2ae71ef3fd1e s-eunuc:5000/dip/mlr-worker:x86_64-latest "/w/w /home/dip/bin/…" 5 seconds ago Up 4 seconds mlr_worker_04 c243e08c084a s-eunuc:5000/dip/mlr-worker:x86_64-latest "/w/w /home/dip/bin/…" 5 seconds ago Up 4 seconds mlr_worker_03 0daf9cc0681d s-eunuc:5000/dip/mlr-worker:x86_64-latest "/w/w /home/dip/bin/…" 5 seconds ago Up 4 seconds mlr_worker_02 6a526b469483 s-eunuc:5000/dip/mlr-worker:x86_64-latest "/w/w /home/dip/bin/…" 5 seconds ago Up 4 seconds mlr_worker_01 896fc980377d s-eunuc:5000/dip/mlr-worker:x86_64-latest "/w/w /home/dip/bin/…" 5 seconds ago Up 4 seconds mlr_worker00

While the containers are active, the weavenet internal DNS allocated valid addresses for the containers:

_➜ WEAVE ~/bin/WEAVE/weave status dns mlr_worker_00 10.32.0.1 896fc980377d ca:ba:37:a3:76:5d mlr_worker_01 10.32.0.2 6a526b469483 ca:ba:37:a3:76:5d mlr_worker_02 10.32.0.4 0daf9cc0681d ca:ba:37:a3:76:5d mlr_worker_03 10.32.0.3 c243e08c084a ca:ba:37:a3:76:5d mlr_worker_04 10.32.0.5 2ae71ef3fd1e ca:ba:37:a3:76:5d mlr_worker_05 10.32.0.8 15473e656da3 ca:ba:37:a3:76:5d mlr_worker_06 10.32.0.9 cb7132a1be3f ca:ba:37:a3:76:5d mlr_worker07 10.32.0.7 129240b4b9bc ca:ba:37:a3:76:5d scope 10.32.0.6 1da739008d9e ca:ba:37:a3:76:5d scope 10.193.224.139 1da739008d9e ca:ba:37:a3:76:5d scope 192.168.2.1 1da739008d9e ca:ba:37:a3:76:5d scope 192.168.3.1 1da739008d9e ca:ba:37:a3:76:5d scope 192.168.100.2 1da739008d9e ca:ba:37:a3:76:5d scope 192.168.122.1 1da739008d9e ca:ba:37:a3:76:5d

Once the the containers have terminated their job, I check that nothing remains. There are no more containers named "mlr_worker_xx":

➜ WEAVE docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 981d90e3bc65 weaveworks/weave:2.5.2 "/home/weave/weaver …" 2 minutes ago Up 2 minutes weave 2d51aae3a84c weaveworks/weaveexec:2.5.2 "data-only" 2 minutes ago Created weavevolumes-2.5.2 a53191d29a81 weaveworks/weaveexec:2.4.1 "data-only" 10 minutes ago Created weavevolumes-2.4.1 f401bedb8815 weaveworks/weaveexec:2.5.1 "data-only" 17 minutes ago Created weavevolumes-2.5.1 4bbf60ccdb71 weaveworks/weavedb:latest "data-only" 17 minutes ago Created weavedb 1da739008d9e weaveworks/scope:1.10.1 "/home/weave/entrypo…" 2 hours ago Up 2 hours weavescope 20a2e60360c0 alpine/socat "socat tcp-listen:44…" 4 hours ago Up 2 hours 0.0.0.0:444->444/tcp ipfireforwarder 6be11d449fe6 registry:2 "/entrypoint.sh /etc…" 4 hours ago Up 2 hours 0.0.0.0:5000->5000/tcp registry

But, when I check the weavenet DNS entries, some hostnames (not always the sames) remain:

_➜ WEAVE ~/bin/WEAVE/weave status dns > mlr_worker_03 10.32.0.3 c243e08c084a ca:ba:37:a3:76:5d mlr_worker_04 10.32.0.5 2ae71ef3fd1e ca:ba:37:a3:76:5d scope 10.32.0.6 1da739008d9e ca:ba:37:a3:76:5d scope 10.193.224.139 1da739008d9e ca:ba:37:a3:76:5d scope 192.168.2.1 1da739008d9e ca:ba:37:a3:76:5d scope 192.168.3.1 1da739008d9e ca:ba:37:a3:76:5d scope 192.168.100.2 1da739008d9e ca:ba:37:a3:76:5d scope 192.168.122.1 1da739008d9e ca:ba:37:a3:76:5d_

The hosts mlrworker should have been removed as the container has gone.

I made exactly the same test with weavenet version V 4.2.1, and all hostnames disappear as soon as the containers are gone.

Consequence of this bug:

When I restart a container with the same name, a second DNS entry is allocated for the new instance, but the DNS resolve inside the running container returns the first allocated address (indeed false).

_➜ WEAVE ~/bin/WEAVE/weave status dns mlr_worker_00 10.32.0.1 2a0060f5de77 ca:ba:37:a3:76:5d mlr_worker_01 10.32.0.5 021436c028f7 ca:ba:37:a3:76:5d mlr_worker_02 10.32.0.2 738b9c45eed5 ca:ba:37:a3:76:5d mlr_worker_03 10.32.0.3 ad3c42a5502f ca:ba:37:a3:76:5d mlr_worker_03 10.32.0.3 c243e08c084a ca:ba:37:a3:76:5d mlr_worker_04 10.32.0.5 2ae71ef3fd1e ca:ba:37:a3:76:5d mlr_worker_04 10.32.0.4 4668b9063c44 ca:ba:37:a3:76:5d mlr_worker_06 10.32.0.7 6c4f8999a8c2 ca:ba:37:a3:76:5d mlr_worker07 10.32.0.8 7bfcbdd1c1db ca:ba:37:a3:76:5d scope 10.32.0.6 1da739008d9e ca:ba:37:a3:76:5d scope 10.193.224.139 1da739008d9e ca:ba:37:a3:76:5d scope 192.168.2.1 1da739008d9e ca:ba:37:a3:76:5d scope 192.168.3.1 1da739008d9e ca:ba:37:a3:76:5d scope 192.168.100.2 1da739008d9e ca:ba:37:a3:76:5d scope 192.168.122.1 1da739008d9e ca:ba:37:a3:76:5d

In my case this leads to a socket hangup while trying to connect.

How to reproduce it?

Anything else we need to know?

Versions:

$ weave version:
2.5.x
$ docker version:
Docker version 18.09.6, build 481bc77156
$ uname -a
Linux s-eunuc 3.10.0-957.21.2.el7.x86_64 #1 SMP Wed Jun 5 14:26:44 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
$ kubectl version

Logs:

$ docker logs weave

or, if using Kubernetes:

$ kubectl logs -n kube-system <weave-net-pod> weave

Network:

$ ip route

> default via 10.193.224.1 dev eno1 proto dhcp metric 101
> 10.32.0.0/12 dev weave proto kernel scope link src 10.32.0.6
> 10.193.224.0/24 dev eno1 proto kernel scope link src 10.193.224.139 metric 101
> 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
> 192.168.2.0/24 dev virbr2 proto kernel scope link src 192.168.2.1
> 192.168.3.0/24 dev virbr1 proto kernel scope link src 192.168.3.1
> 192.168.100.0/24 dev enp0s20f0u3 proto kernel scope link src 192.168.100.2 metric 100
> 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1

$ ip -4 -o addr

> 1: lo    inet 127.0.0.1/8 scope host lo\       valid_lft forever preferred_lft forever
> 2: enp0s20f0u3    inet 192.168.100.2/24 brd 192.168.100.255 scope global noprefixroute enp0s20f0u3\       valid_lft forever preferred_lft forever
> 3: eno1    inet 10.193.224.139/24 brd 10.193.224.255 scope global noprefixroute dynamic eno1\       valid_lft 108814sec preferred_lft 108814sec
> 5: virbr2    inet 192.168.2.1/24 brd 192.168.2.255 scope global virbr2\       valid_lft forever preferred_lft forever
> 7: virbr1    inet 192.168.3.1/24 brd 192.168.3.255 scope global virbr1\       valid_lft forever preferred_lft forever
> 9: virbr0    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0\       valid_lft forever preferred_lft forever
> 2777: weave    inet 10.32.0.6/12 brd 10.47.255.255 scope global weave\       valid_lft forever preferred_lft forever
> 2018: docker0    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0\       valid_lft forever preferred_lft forever

$ 

> # Generated by iptables-save v1.4.21 on Tue Jun 18 15:44:22 2019
> *mangle
> :PREROUTING ACCEPT [33482659:20198499993]
> :INPUT ACCEPT [31801729:18809985923]
> :FORWARD ACCEPT [1311930:1366195182]
> :OUTPUT ACCEPT [27465604:22365051637]
> :POSTROUTING ACCEPT [28777534:23731246819]
> -A POSTROUTING -o virbr0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
> -A POSTROUTING -o virbr1 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
> -A POSTROUTING -o virbr2 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
> COMMIT
> # Completed on Tue Jun 18 15:44:22 2019
> # Generated by iptables-save v1.4.21 on Tue Jun 18 15:44:22 2019
> *nat
> :PREROUTING ACCEPT [2966:235348]
> :INPUT ACCEPT [2918:231668]
> :OUTPUT ACCEPT [5352:372922]
> :POSTROUTING ACCEPT [5125:357894]
> :DOCKER - [0:0]
> :WEAVE - [0:0]
> -A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
> -A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
> -A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
> -A POSTROUTING -s 192.168.122.0/24 -d 224.0.0.0/24 -j RETURN
> -A POSTROUTING -s 192.168.122.0/24 -d 255.255.255.255/32 -j RETURN
> -A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
> -A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
> -A POSTROUTING -s 192.168.122.0/24 ! -d 192.168.122.0/24 -j MASQUERADE
> -A POSTROUTING -s 192.168.3.0/24 -d 224.0.0.0/24 -j RETURN
> -A POSTROUTING -s 192.168.3.0/24 -d 255.255.255.255/32 -j RETURN
> -A POSTROUTING -s 192.168.3.0/24 ! -d 192.168.3.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
> -A POSTROUTING -s 192.168.3.0/24 ! -d 192.168.3.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
> -A POSTROUTING -s 192.168.3.0/24 ! -d 192.168.3.0/24 -j MASQUERADE
> -A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p tcp -m tcp --dport 5000 -j MASQUERADE
> -A POSTROUTING -s 172.17.0.3/32 -d 172.17.0.3/32 -p tcp -m tcp --dport 444 -j MASQUERADE
> -A POSTROUTING -j WEAVE
> -A DOCKER -i docker0 -j RETURN
> -A DOCKER ! -i docker0 -p tcp -m tcp --dport 5000 -j DNAT --to-destination 172.17.0.2:5000
> -A DOCKER ! -i docker0 -p tcp -m tcp --dport 444 -j DNAT --to-destination 172.17.0.3:444
> -A WEAVE -s 10.32.0.0/12 -d 224.0.0.0/4 -j RETURN
> -A WEAVE ! -s 10.32.0.0/12 -d 10.32.0.0/12 -j MASQUERADE
> -A WEAVE -s 10.32.0.0/12 ! -d 10.32.0.0/12 -j MASQUERADE
> COMMIT
> # Completed on Tue Jun 18 15:44:22 2019
> # Generated by iptables-save v1.4.21 on Tue Jun 18 15:44:22 2019
> *filter
> :INPUT ACCEPT [40439:22080644]
> :FORWARD ACCEPT [0:0]
> :OUTPUT ACCEPT [36693:23658643]
> :DOCKER - [0:0]
> :DOCKER-ISOLATION-STAGE-1 - [0:0]
> :DOCKER-ISOLATION-STAGE-2 - [0:0]
> :DOCKER-USER - [0:0]
> :WEAVE-EXPOSE - [0:0]
> -A INPUT -i virbr0 -p udp -m udp --dport 53 -j ACCEPT
> -A INPUT -i virbr0 -p tcp -m tcp --dport 53 -j ACCEPT
> -A INPUT -i virbr0 -p udp -m udp --dport 67 -j ACCEPT
> -A INPUT -i virbr0 -p tcp -m tcp --dport 67 -j ACCEPT
> -A INPUT -i virbr1 -p udp -m udp --dport 53 -j ACCEPT
> -A INPUT -i virbr1 -p tcp -m tcp --dport 53 -j ACCEPT
> -A INPUT -i virbr1 -p udp -m udp --dport 67 -j ACCEPT
> -A INPUT -i virbr1 -p tcp -m tcp --dport 67 -j ACCEPT
> -A INPUT -i virbr2 -p udp -m udp --dport 53 -j ACCEPT
> -A INPUT -i virbr2 -p tcp -m tcp --dport 53 -j ACCEPT
> -A INPUT -i virbr2 -p udp -m udp --dport 67 -j ACCEPT
> -A INPUT -i virbr2 -p tcp -m tcp --dport 67 -j ACCEPT
> -A INPUT -d 172.17.0.1/32 -i docker0 -p tcp -m tcp --dport 6783 -j DROP
> -A INPUT -d 172.17.0.1/32 -i docker0 -p udp -m udp --dport 6783 -j DROP
> -A INPUT -d 172.17.0.1/32 -i docker0 -p udp -m udp --dport 6784 -j DROP
> -A INPUT -i docker0 -p udp -m udp --dport 53 -j ACCEPT
> -A INPUT -i docker0 -p tcp -m tcp --dport 53 -j ACCEPT
> -A FORWARD -j DOCKER-USER
> -A FORWARD -i docker0 -o weave -j DROP
> -A FORWARD -i weave -o weave -j ACCEPT
> -A FORWARD -o weave -j WEAVE-EXPOSE
> -A FORWARD -i weave ! -o weave -j ACCEPT
> -A FORWARD -o weave -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
> -A FORWARD -j DOCKER-ISOLATION-STAGE-1
> -A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
> -A FORWARD -o docker0 -j DOCKER
> -A FORWARD -i docker0 ! -o docker0 -j ACCEPT
> -A FORWARD -i docker0 -o docker0 -j ACCEPT
> -A FORWARD -d 192.168.122.0/24 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
> -A FORWARD -s 192.168.122.0/24 -i virbr0 -j ACCEPT
> -A FORWARD -i virbr0 -o virbr0 -j ACCEPT
> -A FORWARD -o virbr0 -j REJECT --reject-with icmp-port-unreachable
> -A FORWARD -i virbr0 -j REJECT --reject-with icmp-port-unreachable
> -A FORWARD -d 192.168.3.0/24 -o virbr1 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
> -A FORWARD -s 192.168.3.0/24 -i virbr1 -j ACCEPT
> -A FORWARD -i virbr1 -o virbr1 -j ACCEPT
> -A FORWARD -o virbr1 -j REJECT --reject-with icmp-port-unreachable
> -A FORWARD -i virbr1 -j REJECT --reject-with icmp-port-unreachable
> -A FORWARD -i virbr2 -o virbr2 -j ACCEPT
> -A FORWARD -o virbr2 -j REJECT --reject-with icmp-port-unreachable
> -A FORWARD -i virbr2 -j REJECT --reject-with icmp-port-unreachable
> -A OUTPUT -o virbr0 -p udp -m udp --dport 68 -j ACCEPT
> -A OUTPUT -o virbr1 -p udp -m udp --dport 68 -j ACCEPT
> -A OUTPUT -o virbr2 -p udp -m udp --dport 68 -j ACCEPT
> -A DOCKER -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 5000 -j ACCEPT
> -A DOCKER -d 172.17.0.3/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 444 -j ACCEPT
> -A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
> -A DOCKER-ISOLATION-STAGE-1 -j RETURN
> -A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
> -A DOCKER-ISOLATION-STAGE-2 -j RETURN
> -A DOCKER-USER -j RETURN
> -A WEAVE-EXPOSE -d 10.32.0.0/12 -j ACCEPT
> COMMIT
> # Completed on Tue Jun 18 15:44:22 2019
bernhara commented 5 years ago

Since this problem is locking me, I found a simple workaround: I generated a unique hostname for the time the container is run:

DOCKER_HOST=unix:///var/run/weave/weave.sock ORIG_DOCKERHOST= docker run -it --rm --name "a good container name" --hostname "name with generated uid suffix.weave.local"_ hello-world

On Scope, the container appears with the "the good container name"