search

weaveworks / weave

Simple, resilient multi-host containers networking and more.
https://www.weave.works
Apache License 2.0
6.62k stars 670 forks source link

mesos + weave cni issue #2486

Closed osallou closed 8 years ago

osallou commented 8 years ago

Not sure if it is mesos or weave issue. While testing mesos + weave cni, weave assign correctly to the mesos task but mesos task remains and staging then failure with message:

I0824 08:31:32.086134 26525 cni.cpp:716] Bind mounted '/proc/6248/ns/net' to '/run/mesos/isolators/network/cni/5e284273-0a70-4d69-a35b-866335643003/ns' for container 5e284273-0a70-4d69-a35b-866335643003
I0824 08:31:32.568583 26520 cni.cpp:1109] Got assigned IPv4 address '10.32.0.1/12' from CNI network 'weave' for container 5e284273-0a70-4d69-a35b-866335643003
I0824 08:31:32.568788 26520 cni.cpp:838] Unable to find DNS nameservers for container 5e284273-0a70-4d69-a35b-866335643003. Using host '/etc/resolv.conf'

It seems there is a DNS issue.

Weave status gives:

    Version: 1.6.1 (up to date; next check at 2016/08/24 11:10:52)

    Service: router
   Protocol: weave 1..2
       Name: 9e:1e:28:6d:ad:e6(tifenn)
 Encryption: disabled
PeerDiscovery: enabled
    Targets: 0
Connections: 0
      Peers: 1

TrustedSubnets: none

    Service: ipam
     Status: ready
      Range: 10.32.0.0/12
DefaultSubnet: 10.32.0.0/12

    Service: dns
     Domain: weave.local.
   Upstream: 127.0.1.1, 131.254.130.100
        TTL: 1
    Entries: 0

    Service: proxy
    Address: unix:///var/run/weave/weave.sock

    Service: plugin
 DriverName: weave

This is a single configuration (for testing). I did a weave expose on node. Execute sample Mesos execute fails with above messages:

sudo mesos-execute --command="sleep 2" -docker_image=centos:latest --master=127.0.0.1:5050 --name=test0   --networks=weave

Docker (1.12.1) shows:

CONTAINER ID        IMAGE                        COMMAND                  CREATED             STATUS              PORTS               NAMES
6c03c57377d0        weaveworks/plugin:1.6.1      "/home/weave/plugin"     25 minutes ago      Up 25 minutes                           weaveplugin
62ae1dc3b04e        weaveworks/weaveexec:1.6.1   "/home/weave/weavepro"   25 minutes ago      Up 25 minutes                           weaveproxy
5c51ea58dbfb        weaveworks/weave:1.6.1       "/home/weave/weaver -"   25 minutes ago      Up 25 minutes                           weave
osallou commented 8 years ago

Additional info: running a Docker container with --net=weave works fine Routes in containers show weave network

[root@ccb1b4ebca81 /]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.19.0.1      0.0.0.0         UG    0      0        0 eth1
10.32.0.0       0.0.0.0         255.240.0.0     U     0      0        0 ethwe0
172.19.0.0      0.0.0.0         255.255.0.0     U     0      0        0 eth1
224.0.0.0       0.0.0.0         240.0.0.0       U     0      0        0 ethwe0

So it is related to mesos integration.

rade commented 8 years ago

I0824 08:31:32.568788 26520 cni.cpp:838] Unable to find DNS nameservers for container 5e284273-0a70-4d69-a35b-866335643003. Using host '/etc/resolv.conf'

What makes you think this indicates a failure?

osallou commented 8 years ago

I do not know if this is the failing error, but there is no additional log in mesos, and task remains in STAGING and never succeeds when specifying --networks=weave. If I do not set the --networks option, task works nicely. This may be a mesos CNI bug (investigating this way too).

I followed instructions at https://www.weave.works/docs/net/latest/cni-plugin/

osallou commented 8 years ago

seems mesos related, I have same behavior with Calico, closing issue.