search

weaveworks / weave

Simple, resilient multi-host containers networking and more.
https://www.weave.works
Apache License 2.0
6.62k stars 670 forks source link

No Route to host with v2 plugin on Docker Swarm Mode #3120

Closed gcavalcante8808 closed 6 years ago

gcavalcante8808 commented 7 years ago

What you expected to happen?

Iperf3 from client should connect to server, measuring total bandwitch or at least telnet should connect to port 5201.

What happened?

Iperf3 returned:

iperf3: error - unable to connect to server: No route to host.

Using an overlay network, iperf3 runs normally.

On Weave, DNS resolution and ICMP Ping works (with iptables -F before docker service start action)

How to reproduce it?

  1. Install weave plugin and configure to use multicast;
  2. Create docker cluster (all nodes are managers);
  3. Create stack using the following docker-compose file:

version: '3.1'

services:

weave1: image: networkstatic/iperf3 command: -s deploy: placement: constraints:

networks: default: driver: store/weaveworks/net-plugin:latest_release external: false

  1. Access vm2 ssh and try to issue the command docker exec ping weave1 (It Should Work, DNS and ICMP Ping);

  2. Try to use iperf3 to measure bandwitch:

docker exec iperf3 -c weave1

Anything else we need to know?

I'm using KVM on my personal machine with 2 VMs, which has Docker 17.06 on top of Centos 7, configured as cluster (swarm mode).

Tried with multicast support enabled and disabled for the plugin; same results.

Logs:


# docker-runc exec -t 02c7cd96a526ed506307d09dc6b03dec5fcaad4345ba9c17e3077e0263919ca5 sh
# /home/weave/weaver status

INFO: 2017/09/13 11:56:10.885871 Command line options: map[]
INFO: 2017/09/13 11:56:10.885919 weave  git-1da1943df9d8
INFO: 2017/09/13 11:56:10.886230 Bridge type is bridge
INFO: 2017/09/13 11:56:10.886240 Communication between peers is unencrypted.
INFO: 2017/09/13 11:56:10.918688 Our name is c6:49:af:8b:df:aa(vm2.docker.local)
INFO: 2017/09/13 11:56:10.918732 Launch detected - using supplied peer list: [status]
INFO: 2017/09/13 11:56:10.920162 Docker API on unix:///var/run/docker.sock: &[Version=17.06.2-ce ApiVersion=1.30 Os=linux KernelVersion=3.10.0-514.26.2.el7.x86_64 BuildTime=2017-09-05T20:00:25.777242273+00:00 MinAPIVersion=1.12 GitCommit=cec0b72 GoVersion=go1.8.3 Arch=amd64]
INFO: 2017/09/13 11:56:10.920743 Listening for DNS queries on 0.0.0.0:53
INFO: 2017/09/13 11:56:10.943083 Sniffing traffic on vethwe-pcap (via pcap)
FATA: 2017/09/13 11:56:10.956896 listen udp4 :6783: bind: address already in use

Using "weave" instead of "weaver", returns errors:

sh -x /home/weave/weave status

+ set -e
+ [ -n  ]
+ SCRIPT_VERSION=unreleased
+ IMAGE_VERSION=latest
+ [ unreleased = unreleased ]
+ IMAGE_VERSION=latest
+ MIN_DOCKER_VERSION=1.10.0
+ DOCKERHUB_USER=weaveworks
+ BASE_EXEC_IMAGE=weaveworks/weaveexec
+ EXEC_IMAGE=weaveworks/weaveexec:latest
+ WEAVEDB_IMAGE=weaveworks/weavedb
+ echo 
+ cut -s -d: -f1
+ PROXY_HOST=
+ PROXY_HOST=127.0.0.1
+ DOCKER_CLIENT_HOST=
+ IP_REGEXP=[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}
+ CIDR_REGEXP=[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}/[0-9]{1,2}
+ check_docker_version
+ docker -v
/home/weave/weave: line 1: docker: not found+ 
sed -n -e s|^Docker version \([0-9][0-9]*\.[0-9][0-9]*\.[0-9][0-9]*\).*|\1|p
+ DOCKER_VERSION=
+ [ -z  ]
+ echo ERROR: Unable to parse docker version
ERROR: Unable to parse docker version
+ exit 1

Firewall:

iptables -t nat -L

Chain PREROUTING (policy ACCEPT) target prot opt source destination
DOCKER all -- anywhere anywhere ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT) target prot opt source destination

Chain OUTPUT (policy ACCEPT) target prot opt source destination
DOCKER all -- anywhere !loopback/8 ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT) target prot opt source destination
MASQUERADE all -- 172.17.0.0/16 anywhere
MASQUERADE all -- 172.18.0.0/16 anywhere

Chain DOCKER (2 references) target prot opt source destination
RETURN all -- anywhere anywhere
RETURN all -- anywhere anywhere

Chain OUTPUT_direct (0 references) target prot opt source destination

Chain POSTROUTING_ZONES (0 references) target prot opt source destination

Chain POSTROUTING_ZONES_SOURCE (0 references) target prot opt source destination

Chain POSTROUTING_direct (0 references) target prot opt source destination

Chain POST_public (0 references) target prot opt source destination

Chain POST_public_allow (0 references) target prot opt source destination

Chain POST_public_deny (0 references) target prot opt source destination

Chain POST_public_log (0 references) target prot opt source destination

Chain PREROUTING_ZONES (0 references) target prot opt source destination

Chain PREROUTING_ZONES_SOURCE (0 references) target prot opt source destination

Chain PREROUTING_direct (0 references) target prot opt source destination

Chain PRE_public (0 references) target prot opt source destination

Chain PRE_public_allow (0 references) target prot opt source destination

Chain PRE_public_deny (0 references) target prot opt source destination

Chain PRE_public_log (0 references) target prot opt source destination

iptables -L

Chain INPUT (policy ACCEPT) target prot opt source destination

Chain FORWARD (policy ACCEPT) target prot opt source destination
DOCKER-USER all -- anywhere anywhere
DOCKER-ISOLATION all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
DROP all -- anywhere anywhere

Chain OUTPUT (policy ACCEPT) target prot opt source destination

Chain DOCKER (2 references) target prot opt source destination

Chain DOCKER-ISOLATION (1 references) target prot opt source destination
DROP all -- anywhere anywhere
DROP all -- anywhere anywhere
RETURN all -- anywhere anywhere

Chain DOCKER-USER (1 references) target prot opt source destination
RETURN all -- anywhere anywhere

Chain FORWARD_IN_ZONES (0 references) target prot opt source destination

Chain FORWARD_IN_ZONES_SOURCE (0 references) target prot opt source destination

Chain FORWARD_OUT_ZONES (0 references) target prot opt source destination

Chain FORWARD_OUT_ZONES_SOURCE (0 references) target prot opt source destination

Chain FORWARD_direct (0 references) target prot opt source destination

Chain FWDI_public (0 references) target prot opt source destination

Chain FWDI_public_allow (0 references) target prot opt source destination

Chain FWDI_public_deny (0 references) target prot opt source destination

Chain FWDI_public_log (0 references) target prot opt source destination

Chain FWDO_public (0 references) target prot opt source destination

Chain FWDO_public_allow (0 references) target prot opt source destination

Chain FWDO_public_deny (0 references) target prot opt source destination

Chain FWDO_public_log (0 references) target prot opt source destination

Chain INPUT_ZONES (0 references) target prot opt source destination

Chain INPUT_ZONES_SOURCE (0 references) target prot opt source destination

Chain INPUT_direct (0 references) target prot opt source destination

Chain IN_public (0 references) target prot opt source destination

Chain IN_public_allow (0 references) target prot opt source destination

Chain IN_public_deny (0 references) target prot opt source destination

Chain IN_public_log (0 references) target prot opt source destination

Chain OUTPUT_direct (0 references)

TCPDUMP

From container weave2 (that is on vm2) to weave1(that is on vm1):

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ethwe0, link-type EN10MB (Ethernet), capture size 262144 bytes 12:30:07.213779 ARP, Request who-has weave_weave1.1.ubaoxfck11mokt2pdm68r9tup.weave_default tell 48b3ca73500a, length 28 12:30:08.216357 ARP, Request who-has weave_weave1.1.ubaoxfck11mokt2pdm68r9tup.weave_default tell 48b3ca73500a, length 28 12:30:09.218356 ARP, Request who-has weave_weave1.1.ubaoxfck11mokt2pdm68r9tup.weave_default tell 48b3ca73500a, length 28 12:30:10.220434 ARP, Request who-has weave_weave1.1.ubaoxfck11mokt2pdm68r9tup.weave_default tell 48b3ca73500a, length 28 12:30:11.222364 ARP, Request who-has weave_weave1.1.ubaoxfck11mokt2pdm68r9tup.weave_default tell 48b3ca73500a, length 28 12:30:12.224358 ARP, Request who-has weave_weave1.1.ubaoxfck11mokt2pdm68r9tup.weave_default tell 48b3ca73500a, length 28

Thanks in advance.

realcbb commented 7 years ago

I have the same issue with you. Containers on different host cannot ping each other.

ps -ef | grep weave results 

root      3263  3245  0 13:41 ?        00:00:00 /home/weave/weaver --port=6783 --datapath=datapath --host-root=/host --proc-path=/host/proc --http-addr=127.0.0.1:6784 --status-addr=0.0.0.0:6782 --no-dns --ipalloc-range=10.32.0.0/12 --nickname red-hat-enterprise-linux2.shared --log-level=debug --db-prefix=/host/var/lib/weave/weave --plugin-v2 --plugin-v2-multicast --plugin-mesh-socket= --docker-api=

There is no manager node's ip at the end. Maybe this is one problem. The other one is in the status log listen udp4 :6783: bind: address already in use.

bboreham commented 7 years ago

@gcavalcante8808 to run the status command inside the V2 plugin you have to add --local:

/home/weave/weave --local status

were you following some instructions somewhere we should update?

realcbb commented 7 years ago 
/ # /home/weave/weave --local status

        Version: 2.0.1 (up to date; next check at 2017/09/25 15:07:19)

        Service: router
       Protocol: weave 1..2
           Name: be:e6:3c:3d:c3:a5(red-hat-enterprise-linux.shared)
     Encryption: disabled
  PeerDiscovery: enabled
        Targets: 1
    Connections: 1 (1 failed)
          Peers: 1
 TrustedSubnets: none

        Service: ipam
         Status: idle
          Range: 10.32.0.0/12
  DefaultSubnet: 10.32.0.0/12

        Service: plugin (v2)
rade commented 6 years ago

It seems like we've reached a dead end in the investigation here. -> closing