search

weaveworks / weave

Simple, resilient multi-host containers networking and more.
https://www.weave.works
Apache License 2.0
6.62k stars 670 forks source link

Duplicate automated IP address assignment #3003

Closed avkonst closed 7 years ago

avkonst commented 7 years ago

I am executing the following sequence:

weave launch \
        --ipalloc-range 10.32.0.0/13 --ipalloc-default-subnet 10.32.0.0/12 \
        --name=::1 --ipalloc-init seed=::1,::2,::3,::4

docker run --name sleep-client -h sleep-client.weave.local -dti \
        --init \
        --net=weave \
        $(weave dns-args) \
        --restart always \
        ubuntu:16.04 \
        sleep 2000

docker run --name sleep-client2 -h sleep-client2.weave.local -dti \
        --init \
        --net=weave --ip=10.40.1.11 \
        $(weave dns-args) \
        --restart always \
        ubuntu:16.04 \
        sleep 2000

docker run --name sleep-client3 -h sleep-client3.weave.local -dti \
        --init \
        --net=weave \
        $(weave dns-args) \
        --restart always \
        ubuntu:16.04 \
        sleep 2000

What you expected to happen?

The idea is that the allocated IP range is split in half for manual and automated IP address assignment. The first and the third containers should get the IP automatically from the 10.32.0.0/13 range. The second has got IP address explicitly assigned from the reserved range for manual assignment.

What happened?

Duplicate IP address for the second and the third container

"DNS": {
    "Domain": "weave.local.",
    "Upstream": [
      "192.168.178.1"
    ],
    "Address": "172.17.0.1:53",
    "TTL": 1,
    "Entries": [
      {
        "Hostname": "sleep-client.weave.local.",
        "Origin": "00:00:00:00:00:01",
        "ContainerID": "8d1e717444252f01f70f527c40274f1512bc3a605e32302720adb02b5a3d741c",
        "Address": "10.32.0.1",
        "Version": 0,
        "Tombstone": 0
      },
      {
        "Hostname": "sleep-client2.weave.local.",
        "Origin": "00:00:00:00:00:01",
        "ContainerID": "841eb894193b95261fece3397b82d931f53d5788edae707664febb75ae098322",
        "Address": "10.40.1.11",
        "Version": 0,
        "Tombstone": 0
      },
      {
        "Hostname": "sleep-client3.weave.local.",
        "Origin": "00:00:00:00:00:01",
        "ContainerID": "0ade68c5b9816f7e32122a4b66643a505f6f4b8c5914464942528d947961393c",
        "Address": "10.40.1.11",
        "Version": 0,
        "Tombstone": 0
      }
    ]
  }

How to reproduce it?

The script above

Anything else we need to know?

Environment: virtualbox VM.

Versions:

See with weave 1.9.5 and 1.9.7 docker version 1.13.1

marccarre commented 7 years ago

Hi @avkonst, thank you very much for reporting this, it does indeed look like a bug!

marccarre commented 7 years ago

With:

$ docker run --rm --name c1 -h c1.weave.local $(weave dns-args) -d \
    --net=weave \
    alpine tail -f /dev/null

$ docker run --rm --name c2 -h c2.weave.local $(weave dns-args) -d \
    --net=weave --ip=10.40.1.11 \
    alpine tail -f /dev/null

$ docker run --rm --name c3 -h c3.weave.local $(weave dns-args) -d \
    --net=weave \
    alpine tail -f /dev/null

I am able to reproduce the issue in scenario 1 below:

1)

$ weave launch --ipalloc-range 10.32.0.0/13 --ipalloc-default-subnet 10.32.0.0/12
$ weave status dns
c1           10.32.0.1       84098daec7f9 3a:69:c9:53:af:9e
c2           10.40.1.11      92b91f6c5d8c 3a:69:c9:53:af:9e
c3           10.40.1.11      aa5a69967a33 3a:69:c9:53:af:9e

2)

$ weave launch
$ weave status dns
c1           10.32.0.1       eb0a8b1d3be1 3a:69:c9:53:af:9e
c2           10.40.1.11      8e9455475176 3a:69:c9:53:af:9e
c3           10.32.0.2       ad9e0e1eb0d5 3a:69:c9:53:af:9e

3)

$ weave launch --ipalloc-range 10.32.0.0/12
$ weave status dns
c1           10.32.0.1       c8d85261a082 3a:69:c9:53:af:9e
c2           10.40.1.11      8e265296e1c7 3a:69:c9:53:af:9e
c3           10.32.0.2       0aa95f3bb8f8 3a:69:c9:53:af:9e

Output of weave report:

$ weave report
{
    "Version": "1.9.7",
    "VersionCheck": {
        "Enabled": true,
        "Success": true,
        "NewVersion": "",
        "NextCheckAt": "2017-06-06T22:31:09.282253946Z"
    },
    "Router": {
        "Protocol": "weave",
        "ProtocolMinVersion": 1,
        "ProtocolMaxVersion": 2,
        "Encryption": false,
        "PeerDiscovery": true,
        "Name": "3a:69:c9:53:af:9e",
        "NickName": "sebulba",
        "Port": 6783,
        "Peers": [
            {
                "Name": "3a:69:c9:53:af:9e",
                "NickName": "sebulba",
                "UID": 4756256240405131759,
                "ShortID": 2119,
                "Version": 0,
                "Connections": null
            }
        ],
        "UnicastRoutes": [
            {
                "Dest": "3a:69:c9:53:af:9e",
                "Via": "00:00:00:00:00:00"
            }
        ],
        "BroadcastRoutes": [
            {
                "Source": "3a:69:c9:53:af:9e",
                "Via": null
            }
        ],
        "Connections": null,
        "TerminationCount": 0,
        "Targets": null,
        "OverlayDiagnostics": {
            "fastdp": {
                "Vports": [
                    {
                        "ID": 0,
                        "Name": "datapath",
                        "TypeName": "internal"
                    },
                    {
                        "ID": 1,
                        "Name": "vethwe-datapath",
                        "TypeName": "netdev"
                    },
                    {
                        "ID": 2,
                        "Name": "vxlan-6784",
                        "TypeName": "vxlan"
                    }
                ],
                "Flows": [
                    {
                        "FlowKeys": [
                            "UnknownFlowKey{type: 23, key: 0000, mask: 0000}",
                            "UnknownFlowKey{type: 24, key: 00000000, mask: 00000000}",
                            "InPortFlowKey{vport: 1}",
                            "UnknownFlowKey{type: 25, key: 00000000000000000000000000000000, mask: 00000000000000000000000000000000}",
                            "EthernetFlowKey{src: 6a:eb:37:72:f7:8a, dst: 33:33:00:00:00:fb}",
                            "UnknownFlowKey{type: 22, key: 00000000, mask: 00000000}"
                        ],
                        "Actions": [
                            "OutputAction{vport: 0}"
                        ],
                        "Packets": 0,
                        "Bytes": 0,
                        "Used": 0
                    },
                    {
                        "FlowKeys": [
                            "InPortFlowKey{vport: 0}",
                            "UnknownFlowKey{type: 22, key: 00000000, mask: 00000000}",
                            "UnknownFlowKey{type: 24, key: 00000000, mask: 00000000}",
                            "EthernetFlowKey{src: 2a:1f:2a:cd:d3:c9, dst: 33:33:00:00:00:fb}",
                            "UnknownFlowKey{type: 23, key: 0000, mask: 0000}",
                            "UnknownFlowKey{type: 25, key: 00000000000000000000000000000000, mask: 00000000000000000000000000000000}"
                        ],
                        "Actions": [
                            "OutputAction{vport: 1}"
                        ],
                        "Packets": 0,
                        "Bytes": 0,
                        "Used": 0
                    }
                ]
            },
            "sleeve": null
        },
        "TrustedSubnets": [],
        "Interface": "datapath (via ODP)",
        "CaptureStats": {
            "FlowMisses": 12
        },
        "MACs": [
            {
                "Mac": "6a:eb:37:72:f7:8a",
                "Name": "3a:69:c9:53:af:9e",
                "NickName": "sebulba",
                "LastSeen": "2017-06-06T15:57:48.525471192Z"
            },
            {
                "Mac": "2a:1f:2a:cd:d3:c9",
                "Name": "3a:69:c9:53:af:9e",
                "NickName": "sebulba",
                "LastSeen": "2017-06-06T15:57:48.525916894Z"
            },
            {
                "Mac": "3a:69:c9:53:af:9e",
                "Name": "3a:69:c9:53:af:9e",
                "NickName": "sebulba",
                "LastSeen": "2017-06-06T15:52:48.526783773Z"
            },
            {
                "Mac": "36:7a:49:cd:34:51",
                "Name": "3a:69:c9:53:af:9e",
                "NickName": "sebulba",
                "LastSeen": "2017-06-06T15:48:03.578934308Z"
            },
            {
                "Mac": "a2:19:22:eb:50:d4",
                "Name": "3a:69:c9:53:af:9e",
                "NickName": "sebulba",
                "LastSeen": "2017-06-06T15:48:07.844696381Z"
            },
            {
                "Mac": "ae:3f:89:d0:39:33",
                "Name": "3a:69:c9:53:af:9e",
                "NickName": "sebulba",
                "LastSeen": "2017-06-06T15:48:13.539768438Z"
            }
        ]
    },
    "IPAM": {
        "Paxos": null,
        "Range": "10.32.0.0/13",
        "RangeNumIPs": 524288,
        "ActiveIPs": 1,
        "DefaultSubnet": "10.32.0.0/12",
        "Entries": [
            {
                "Token": "10.32.0.0",
                "Size": 524288,
                "Peer": "3a:69:c9:53:af:9e",
                "Nickname": "sebulba",
                "IsKnownPeer": true,
                "Version": 1
            }
        ],
        "PendingClaims": null,
        "PendingAllocates": null
    },
    "DNS": {
        "Domain": "weave.local.",
        "Upstream": [
            "127.0.1.1"
        ],
        "Address": "172.17.0.1:53",
        "TTL": 1,
        "Entries": [
            {
                "Hostname": "c1.weave.local.",
                "Origin": "3a:69:c9:53:af:9e",
                "ContainerID": "6581a7b75a825e3200912089f2fdeb528a901b9b7c9c72732afbe7983ed19023",
                "Address": "10.32.0.1",
                "Version": 0,
                "Tombstone": 0
            },
            {
                "Hostname": "c2.weave.local.",
                "Origin": "3a:69:c9:53:af:9e",
                "ContainerID": "78edaa248030b82b54f90cfde6bcdd435c0431052750a646b69c5a4bc28c6124",
                "Address": "10.40.1.11",
                "Version": 0,
                "Tombstone": 0
            },
            {
                "Hostname": "c3.weave.local.",
                "Origin": "3a:69:c9:53:af:9e",
                "ContainerID": "964a64f63134d73759a6f709779216dfba675dacf98503c8eeba0df24f594344",
                "Address": "10.40.1.11",
                "Version": 0,
                "Tombstone": 0
            }
        ]
    }
}

weave's logs:

$ docker logs weave
INFO: 2017/06/06 15:47:48.503652 Command line options: map[log-level:debug nickname:sebulba port:6783 status-addr:127.0.0.1:6782 datapath:datapath ipalloc-range:10.32.0.0/13 ipalloc-default-subnet:10.32.0.0/12 name:3a:69:c9:53:af:9e resolv-conf:/var/run/weave/etc/resolv.conf dns-effective-listen-address:172.17.0.1 dns-listen-address:172.17.0.1:53 http-addr:127.0.0.1:6784]
INFO: 2017/06/06 15:47:48.503688 Communication between peers is unencrypted.
DEBU: 2017/06/06 15:47:48.517139 fastdp: broadcast{0 false} {2a:1f:2a:cd:d3:c9 33:33:00:00:00:16}
DEBU: 2017/06/06 15:47:48.517367 Creating ODP flow FlowSpec{keys: [EthernetFlowKey{src: 2a:1f:2a:cd:d3:c9, dst: 33:33:00:00:00:16} InPortFlowKey{vport: 0}], actions: [OutputAction{vport: 1}]}
INFO: 2017/06/06 15:47:48.531421 Our name is 3a:69:c9:53:af:9e(sebulba)
INFO: 2017/06/06 15:47:48.531595 Launch detected - using supplied peer list: []
INFO: 2017/06/06 15:47:48.538412 Docker API on unix:///var/run/docker.sock: &[GitCommit=89658be Arch=amd64 Version=17.05.0-ce ApiVersion=1.29 Os=linux KernelVersion=4.4.0-78-generic BuildTime=2017-05-04T22:10:54.638119411+00:00 MinAPIVersion=1.12 GoVersion=go1.7.5]
INFO: 2017/06/06 15:47:48.553536 Checking for pre-existing addresses on weave bridge
INFO: 2017/06/06 15:47:48.618715 [allocator 3a:69:c9:53:af:9e] No valid persisted data
INFO: 2017/06/06 15:47:48.631266 [allocator 3a:69:c9:53:af:9e] Initialising via deferred consensus
INFO: 2017/06/06 15:47:48.631375 Listening for DNS queries on 172.17.0.1
INFO: 2017/06/06 15:47:48.631391 Sniffing traffic on datapath (via ODP)
INFO: 2017/06/06 15:47:48.636704 Listening for HTTP control messages on 127.0.0.1:6784
INFO: 2017/06/06 15:47:48.636958 Listening for metrics requests on 127.0.0.1:6782
DEBU: 2017/06/06 15:47:48.693475 [http] GET /status
DEBU: 2017/06/06 15:47:48.760710 fastdp: broadcast{1 false} {6a:eb:37:72:f7:8a 33:33:00:00:00:16}
INFO: 2017/06/06 15:47:48.760741 Discovered local MAC 6a:eb:37:72:f7:8a
DEBU: 2017/06/06 15:47:48.760802 Creating ODP flow FlowSpec{keys: [EthernetFlowKey{src: 6a:eb:37:72:f7:8a, dst: 33:33:00:00:00:16} InPortFlowKey{vport: 1}], actions: [OutputAction{vport: 0}]}
DEBU: 2017/06/06 15:47:48.964955 fastdp: broadcast{0 false} {2a:1f:2a:cd:d3:c9 33:33:00:00:00:02}
INFO: 2017/06/06 15:47:48.965005 Discovered local MAC 2a:1f:2a:cd:d3:c9
DEBU: 2017/06/06 15:47:48.965090 Creating ODP flow FlowSpec{keys: [EthernetFlowKey{src: 2a:1f:2a:cd:d3:c9, dst: 33:33:00:00:00:02} InPortFlowKey{vport: 0}], actions: [OutputAction{vport: 1}]}
DEBU: 2017/06/06 15:47:49.066266 fastdp: broadcast{0 false} {2a:1f:2a:cd:d3:c9 33:33:00:00:00:fb}
DEBU: 2017/06/06 15:47:49.066423 Creating ODP flow FlowSpec{keys: [EthernetFlowKey{src: 2a:1f:2a:cd:d3:c9, dst: 33:33:00:00:00:fb} InPortFlowKey{vport: 0}], actions: [OutputAction{vport: 1}]}
DEBU: 2017/06/06 15:47:49.076351 [http] GET /ipinfo/tracker
DEBU: 2017/06/06 15:47:49.112956 fastdp: broadcast{1 false} {3a:69:c9:53:af:9e 33:33:00:00:00:16}
INFO: 2017/06/06 15:47:49.112992 Discovered local MAC 3a:69:c9:53:af:9e
DEBU: 2017/06/06 15:47:49.113076 Creating ODP flow FlowSpec{keys: [EthernetFlowKey{src: 3a:69:c9:53:af:9e, dst: 33:33:00:00:00:16} InPortFlowKey{vport: 1}], actions: [OutputAction{vport: 0}]}
DEBU: 2017/06/06 15:47:49.113141 fastdp: broadcast{1 false} {3a:69:c9:53:af:9e 33:33:00:00:00:02}
DEBU: 2017/06/06 15:47:49.113206 Creating ODP flow FlowSpec{keys: [EthernetFlowKey{src: 3a:69:c9:53:af:9e, dst: 33:33:00:00:00:02} InPortFlowKey{vport: 1}], actions: [OutputAction{vport: 0}]}
DEBU: 2017/06/06 15:47:49.210922 fastdp: broadcast{1 false} {3a:69:c9:53:af:9e 33:33:00:00:00:fb}
DEBU: 2017/06/06 15:47:49.211167 Creating ODP flow FlowSpec{keys: [EthernetFlowKey{src: 3a:69:c9:53:af:9e, dst: 33:33:00:00:00:fb} InPortFlowKey{vport: 1}], actions: [OutputAction{vport: 0}]}
DEBU: 2017/06/06 15:47:49.319213 fastdp: broadcast{1 false} {6a:eb:37:72:f7:8a 33:33:00:00:00:02}
DEBU: 2017/06/06 15:47:49.319494 Creating ODP flow FlowSpec{keys: [EthernetFlowKey{src: 6a:eb:37:72:f7:8a, dst: 33:33:00:00:00:02} InPortFlowKey{vport: 1}], actions: [OutputAction{vport: 0}]}
DEBU: 2017/06/06 15:47:49.422047 fastdp: broadcast{1 false} {6a:eb:37:72:f7:8a 33:33:00:00:00:fb}
DEBU: 2017/06/06 15:47:49.422356 Creating ODP flow FlowSpec{keys: [EthernetFlowKey{src: 6a:eb:37:72:f7:8a, dst: 33:33:00:00:00:fb} InPortFlowKey{vport: 1}], actions: [OutputAction{vport: 0}]}
DEBU: 2017/06/06 15:47:49.616336 [http] GET /status
DEBU: 2017/06/06 15:47:49.974983 [http] GET /ipinfo/defaultsubnet
DEBU: 2017/06/06 15:48:02.946191 [http] GET /domain
DEBU: 2017/06/06 15:48:03.165580 [http] POST /ip/_/10.32.0.0/12
INFO: 2017/06/06 15:48:03.165697 Assuming quorum size of 1
DEBU: 2017/06/06 15:48:03.165716 [allocator 3a:69:c9:53:af:9e] Paxos proposing
DEBU: 2017/06/06 15:48:03.165772 [allocator 3a:69:c9:53:af:9e]: Paxos consensus: [3a:69:c9:53:af:9e]
DEBU: 2017/06/06 15:48:03.174138 [allocator 3a:69:c9:53:af:9e]: Allocated 10.32.0.1 for 10.32.0.1 in 10.32.0.0/12
DEBU: 2017/06/06 15:48:03.571871 [http] PUT /name/6581a7b75a825e3200912089f2fdeb528a901b9b7c9c72732afbe7983ed19023/10.32.0.1
INFO: 2017/06/06 15:48:03.572079 [nameserver 3a:69:c9:53:af:9e] adding entry for 6581a7b75a825e3200912089f2fdeb528a901b9b7c9c72732afbe7983ed19023: c1.weave.local. -> 10.32.0.1
DEBU: 2017/06/06 15:48:03.578898 fastdp: broadcast{1 false} {36:7a:49:cd:34:51 ff:ff:ff:ff:ff:ff}
INFO: 2017/06/06 15:48:03.578942 Discovered local MAC 36:7a:49:cd:34:51
DEBU: 2017/06/06 15:48:03.579014 Creating ODP flow FlowSpec{keys: [EthernetFlowKey{src: 36:7a:49:cd:34:51, dst: ff:ff:ff:ff:ff:ff} InPortFlowKey{vport: 1}], actions: [OutputAction{vport: 0}]}
DEBU: 2017/06/06 15:48:07.187327 [http] GET /domain
DEBU: 2017/06/06 15:48:07.459765 [http] PUT /ip/_/10.40.1.11/12
INFO: 2017/06/06 15:48:07.459836 [allocator 3a:69:c9:53:af:9e] Address 10.40.1.11/12 claimed by _ - not in our range
DEBU: 2017/06/06 15:48:07.839885 [http] PUT /name/78edaa248030b82b54f90cfde6bcdd435c0431052750a646b69c5a4bc28c6124/10.40.1.11
INFO: 2017/06/06 15:48:07.840014 [nameserver 3a:69:c9:53:af:9e] adding entry for 78edaa248030b82b54f90cfde6bcdd435c0431052750a646b69c5a4bc28c6124: c2.weave.local. -> 10.40.1.11
DEBU: 2017/06/06 15:48:07.844667 fastdp: broadcast{1 false} {a2:19:22:eb:50:d4 ff:ff:ff:ff:ff:ff}
INFO: 2017/06/06 15:48:07.844703 Discovered local MAC a2:19:22:eb:50:d4
DEBU: 2017/06/06 15:48:07.844763 Creating ODP flow FlowSpec{keys: [EthernetFlowKey{src: a2:19:22:eb:50:d4, dst: ff:ff:ff:ff:ff:ff} InPortFlowKey{vport: 1}], actions: [OutputAction{vport: 0}]}
DEBU: 2017/06/06 15:48:12.786624 [http] GET /domain
DEBU: 2017/06/06 15:48:13.071245 [http] POST /ip/_/10.32.0.0/12
DEBU: 2017/06/06 15:48:13.531405 [http] PUT /name/964a64f63134d73759a6f709779216dfba675dacf98503c8eeba0df24f594344/10.40.1.11
INFO: 2017/06/06 15:48:13.531533 [nameserver 3a:69:c9:53:af:9e] adding entry for 964a64f63134d73759a6f709779216dfba675dacf98503c8eeba0df24f594344: c3.weave.local. -> 10.40.1.11
DEBU: 2017/06/06 15:48:13.539725 fastdp: broadcast{1 false} {ae:3f:89:d0:39:33 ff:ff:ff:ff:ff:ff}
INFO: 2017/06/06 15:48:13.539777 Discovered local MAC ae:3f:89:d0:39:33
DEBU: 2017/06/06 15:48:13.539855 Creating ODP flow FlowSpec{keys: [EthernetFlowKey{src: ae:3f:89:d0:39:33, dst: ff:ff:ff:ff:ff:ff} InPortFlowKey{vport: 1}], actions: [OutputAction{vport: 0}]}
DEBU: 2017/06/06 15:48:17.187621 [http] GET /status/dns

This line especially looks suspicious to me: DEBU: 2017/06/06 15:48:13.071245 [http] POST /ip/_/10.32.0.0/12

weaveplugin's logs:

$ docker logs weaveplugin
INFO: 2017/06/06 15:47:49.832978 Weave plugin 1.9.7 Command line options: [--log-level=debug]
INFO: 2017/06/06 15:47:49.833898 Docker API on unix:///var/run/docker.sock: &[Version=17.05.0-ce KernelVersion=4.4.0-78-generic BuildTime=2017-05-04T22:10:54.638119411+00:00 ApiVersion=1.29 MinAPIVersion=1.12 GitCommit=89658be GoVersion=go1.7.5 Os=linux Arch=amd64]
INFO: 2017/06/06 15:47:49.834138 Listening on /run/docker/plugins/weave.sock for global scope
INFO: 2017/06/06 15:47:49.834257 Listening on /run/docker/plugins/weavemesh.sock for local scope
DEBU: 2017/06/06 15:47:49.848150 [net] ContainerStarted: 867b7dd492e4b65b31fa92ce474b5f7a831b59ac5d5fb01f4436e447df89d073
DEBU: 2017/06/06 15:47:49.850057 [net] ContainerStarted: 867b7dd492e4b65b31fa92ce474b5f7a831b59ac5d5fb01f4436e447df89d073
DEBU: 2017/06/06 15:48:02.892422 [net] ContainerStarted: 997b2cbaf7955a7d403868ae91ff9454681474ab199e2165f373fc88907a2776
DEBU: 2017/06/06 15:48:02.896492 [net] ContainerStarted: 997b2cbaf7955a7d403868ae91ff9454681474ab199e2165f373fc88907a2776
INFO: 2017/06/06 15:48:03.164768 [ipam] RequestAddress weave-10.32.0.0/12-10.32.0.0/12 <nil> map[]
DEBU: 2017/06/06 15:48:03.164825 weave POST to http://127.0.0.1:6784/ip/_/10.32.0.0/12 with map[]
DEBU: 2017/06/06 15:48:03.178957 [ipam] RequestAddress result 10.32.0.1/12
DEBU: 2017/06/06 15:48:03.179606 [net] CreateEndpoint &{NetworkID:4ea87a315f089c8db2aacb33bed7f774bcfc23667d300fc3a968cf9c0a464704 EndpointID:a3dc79dafdfa0a603a30c2702e829804e03f23cdbf4b9fd24edd310abc7f5f8d Interface:0xc420275170 Options:map[com.docker.network.endpoint.dnsservers:[172.17.0.1] com.docker.network.endpoint.exposedports:[] com.docker.network.portmap:[]]}
INFO: 2017/06/06 15:48:03.179619 [net] CreateEndpoint a3dc79dafdfa0a603a30c2702e829804e03f23cdbf4b9fd24edd310abc7f5f8d
DEBU: 2017/06/06 15:48:03.179629 interface &{Address:10.32.0.1/12 AddressIPv6: MacAddress:}
DEBU: 2017/06/06 15:48:03.180514 [net] CreateEndpoint &{Response:{Err:} Interface:0xc420275a70}
DEBU: 2017/06/06 15:48:03.203821 [net] JoinEndpoint &{NetworkID:4ea87a315f089c8db2aacb33bed7f774bcfc23667d300fc3a968cf9c0a464704 EndpointID:a3dc79dafdfa0a603a30c2702e829804e03f23cdbf4b9fd24edd310abc7f5f8d SandboxKey:/var/run/docker/netns/67b3f1bba1d6 Options:map[com.docker.network.endpoint.exposedports:[] com.docker.network.portmap:[]]}
INFO: 2017/06/06 15:48:03.203842 [net] JoinEndpoint 4ea87a315f089c8db2aacb33bed7f774bcfc23667d300fc3a968cf9c0a464704:a3dc79dafdfa0a603a30c2702e829804e03f23cdbf4b9fd24edd310abc7f5f8d to /var/run/docker/netns/67b3f1bba1d6
DEBU: 2017/06/06 15:48:03.206693 [net] JoinEndpoint &{Response:{Err:} InterfaceName:0xc4202b00f0 Gateway: GatewayIPv6: StaticRoutes:[{Destination:224.0.0.0/4 RouteType:1 NextHop:}] DisableGatewayService:false}
DEBU: 2017/06/06 15:48:03.569851 [net] ContainerStarted: 6581a7b75a825e3200912089f2fdeb528a901b9b7c9c72732afbe7983ed19023
DEBU: 2017/06/06 15:48:03.569904 [net] ContainerStarted: 6581a7b75a825e3200912089f2fdeb528a901b9b7c9c72732afbe7983ed19023
DEBU: 2017/06/06 15:48:03.571582 weave PUT to http://127.0.0.1:6784/name/6581a7b75a825e3200912089f2fdeb528a901b9b7c9c72732afbe7983ed19023/10.32.0.1 with map[fqdn:[c1.weave.local.]]
DEBU: 2017/06/06 15:48:07.136697 [net] ContainerStarted: 6c926ed5a48db44976faee7e0f562e59ea45d0cacdb0f8462da65d42d93bdc78
DEBU: 2017/06/06 15:48:07.140896 [net] ContainerStarted: 6c926ed5a48db44976faee7e0f562e59ea45d0cacdb0f8462da65d42d93bdc78
INFO: 2017/06/06 15:48:07.459569 [ipam] RequestAddress weave-10.32.0.0/12-10.32.0.0/12 10.40.1.11 map[]
DEBU: 2017/06/06 15:48:07.459595 weave PUT to http://127.0.0.1:6784/ip/_/10.40.1.11/12 with map[]
DEBU: 2017/06/06 15:48:07.463985 [ipam] RequestAddress result 10.40.1.11/12
DEBU: 2017/06/06 15:48:07.464278 [net] CreateEndpoint &{NetworkID:4ea87a315f089c8db2aacb33bed7f774bcfc23667d300fc3a968cf9c0a464704 EndpointID:ad33403100fc8ccd1a9ca1accde0fb21b14e6b3e00a8fda07e529c1ec792a85a Interface:0xc4202ade60 Options:map[com.docker.network.endpoint.dnsservers:[172.17.0.1] com.docker.network.endpoint.exposedports:[] com.docker.network.portmap:[]]}
INFO: 2017/06/06 15:48:07.464287 [net] CreateEndpoint ad33403100fc8ccd1a9ca1accde0fb21b14e6b3e00a8fda07e529c1ec792a85a
DEBU: 2017/06/06 15:48:07.464303 interface &{Address:10.40.1.11/12 AddressIPv6: MacAddress:}
DEBU: 2017/06/06 15:48:07.465099 [net] CreateEndpoint &{Response:{Err:} Interface:0xc420300690}
DEBU: 2017/06/06 15:48:07.484857 [net] JoinEndpoint &{NetworkID:4ea87a315f089c8db2aacb33bed7f774bcfc23667d300fc3a968cf9c0a464704 EndpointID:ad33403100fc8ccd1a9ca1accde0fb21b14e6b3e00a8fda07e529c1ec792a85a SandboxKey:/var/run/docker/netns/134cbf2abf44 Options:map[com.docker.network.endpoint.exposedports:[] com.docker.network.portmap:[]]}
INFO: 2017/06/06 15:48:07.484876 [net] JoinEndpoint 4ea87a315f089c8db2aacb33bed7f774bcfc23667d300fc3a968cf9c0a464704:ad33403100fc8ccd1a9ca1accde0fb21b14e6b3e00a8fda07e529c1ec792a85a to /var/run/docker/netns/134cbf2abf44
DEBU: 2017/06/06 15:48:07.484900 [net] JoinEndpoint &{Response:{Err:} InterfaceName:0xc420300930 Gateway: GatewayIPv6: StaticRoutes:[{Destination:224.0.0.0/4 RouteType:1 NextHop:}] DisableGatewayService:false}
DEBU: 2017/06/06 15:48:07.837416 [net] ContainerStarted: 78edaa248030b82b54f90cfde6bcdd435c0431052750a646b69c5a4bc28c6124
DEBU: 2017/06/06 15:48:07.837850 [net] ContainerStarted: 78edaa248030b82b54f90cfde6bcdd435c0431052750a646b69c5a4bc28c6124
DEBU: 2017/06/06 15:48:07.839640 weave PUT to http://127.0.0.1:6784/name/78edaa248030b82b54f90cfde6bcdd435c0431052750a646b69c5a4bc28c6124/10.40.1.11 with map[fqdn:[c2.weave.local.]]
DEBU: 2017/06/06 15:48:12.746875 [net] ContainerStarted: 3a1c4afb531bea6bdb236488d0202a128cdf90b7e41c8052007f994600f1681c
DEBU: 2017/06/06 15:48:12.746875 [net] ContainerStarted: 3a1c4afb531bea6bdb236488d0202a128cdf90b7e41c8052007f994600f1681c
INFO: 2017/06/06 15:48:13.070527 [ipam] RequestAddress weave-10.32.0.0/12-10.32.0.0/12 <nil> map[]
DEBU: 2017/06/06 15:48:13.070641 weave POST to http://127.0.0.1:6784/ip/_/10.32.0.0/12 with map[]
DEBU: 2017/06/06 15:48:13.072269 [ipam] RequestAddress result 10.40.1.11/12
DEBU: 2017/06/06 15:48:13.073814 [net] CreateEndpoint &{NetworkID:4ea87a315f089c8db2aacb33bed7f774bcfc23667d300fc3a968cf9c0a464704 EndpointID:fc7f70832a49e916b93a3067ae7cea87420c8e21c3cf535d4dcb8d514fbf2a57 Interface:0xc420352b10 Options:map[com.docker.network.endpoint.dnsservers:[172.17.0.1] com.docker.network.endpoint.exposedports:[] com.docker.network.portmap:[]]}
INFO: 2017/06/06 15:48:13.073890 [net] CreateEndpoint fc7f70832a49e916b93a3067ae7cea87420c8e21c3cf535d4dcb8d514fbf2a57
DEBU: 2017/06/06 15:48:13.073941 interface &{Address:10.40.1.11/12 AddressIPv6: MacAddress:}
DEBU: 2017/06/06 15:48:13.086034 [net] CreateEndpoint &{Response:{Err:} Interface:0xc420353320}
DEBU: 2017/06/06 15:48:13.122396 [net] JoinEndpoint &{NetworkID:4ea87a315f089c8db2aacb33bed7f774bcfc23667d300fc3a968cf9c0a464704 EndpointID:fc7f70832a49e916b93a3067ae7cea87420c8e21c3cf535d4dcb8d514fbf2a57 SandboxKey:/var/run/docker/netns/47fabfbea0db Options:map[com.docker.network.endpoint.exposedports:[] com.docker.network.portmap:[]]}
INFO: 2017/06/06 15:48:13.122427 [net] JoinEndpoint 4ea87a315f089c8db2aacb33bed7f774bcfc23667d300fc3a968cf9c0a464704:fc7f70832a49e916b93a3067ae7cea87420c8e21c3cf535d4dcb8d514fbf2a57 to /var/run/docker/netns/47fabfbea0db
DEBU: 2017/06/06 15:48:13.122450 [net] JoinEndpoint &{Response:{Err:} InterfaceName:0xc4203535c0 Gateway: GatewayIPv6: StaticRoutes:[{Destination:224.0.0.0/4 RouteType:1 NextHop:}] DisableGatewayService:false}
DEBU: 2017/06/06 15:48:13.528906 [net] ContainerStarted: 964a64f63134d73759a6f709779216dfba675dacf98503c8eeba0df24f594344
DEBU: 2017/06/06 15:48:13.530469 weave PUT to http://127.0.0.1:6784/name/964a64f63134d73759a6f709779216dfba675dacf98503c8eeba0df24f594344/10.40.1.11 with map[fqdn:[c3.weave.local.]]
DEBU: 2017/06/06 15:48:13.531182 [net] ContainerStarted: 964a64f63134d73759a6f709779216dfba675dacf98503c8eeba0df24f594344
DEBU: 2017/06/06 15:48:17.154061 [net] ContainerStarted: 85574333d29316783016cc1ecb90ed59463fd29d48b19c5d0e66cabde22f8c1d
DEBU: 2017/06/06 15:48:17.154663 [net] ContainerStarted: 85574333d29316783016cc1ecb90ed59463fd29d48b19c5d0e66cabde22f8c1d
bboreham commented 7 years ago

It's because the plugin doesn't have a container or endpoint ID to file the address under, so we pass a NoContainerID magic string. This ID should never get stored, but there's a bug in the logic for addresses outside of the allocation range that means it does get used, and so the next time you ask for an address it finds one filed under that ID and returns it.

alessandrotrisolini commented 7 years ago

Hi everyone!

I reproduced the scenario described by @avkonst and I obtained the same behavior.

I also tried with this:

$ weave launch --ipalloc-range 10.32.0.0/13 --ipalloc-default-subnet 10.32.0.0/12

and

$ docker run --rm --name c1 -h c1.weave.local $(weave dns-args) -d \
    --net=weave --ip=10.40.1.11 \
    alpine tail -f /dev/null

$ docker run --rm --name c2 -h c2.weave.local $(weave dns-args) -d \
    --net=weave \
    alpine tail -f /dev/null

$ docker run --rm --name c3 -h c3.weave.local $(weave dns-args) -d \
    --net=weave \
    alpine tail -f /dev/null

The output of weave status dns is

$ weave status dns
c1           10.40.1.1       7c669ffde349 36:33:c2:02:73:63
c2           10.40.1.1       c5cc6f2b6edc 36:33:c2:02:73:63
c3           10.40.1.1       6b4f98ea7eab 36:33:c2:02:73:63

I also tried with ten containers and they all have the same IP address. It seems that the allocator get stuck when we claim for an IP that is not in the range specified in --ipalloc-range.

If we do a GET request to http://127.0.0.1:6784/ip we obtain this:

$ curl -sS http://127.0.0.1:6784/ip | jq
{
  "owned": [
    {
      "containerid": "_",
      "addrs": [
        "10.40.1.1/12"
      ]
    }
  ]
}

There is another thing that I would like to understand: in the RequestPool method, the iprange variable is initialized with the value of the subnet variable. After doing this, a poolname is created and it will be used in the future as poolID: in this specific case, its value is weave-10.32.0.0/12-10.32.0.0/12, which I think is wrong because the address range is 10.32.0.0/13 and the subnet is 10.32.0.0/12.

When you request an address, the RequestAddress method relies on the poolID in order to retrieve the address range that will be used by the allocator, am I right?

So why iprange and subnet variables are the same in this case? And why is the subPool variable in RequestPool an empty string (I deduced it from the log and because iprange and subnet variables are the same)?

Thank you a lot!

marccarre commented 7 years ago

3010 fixes this in 1.9 and master (via a5d7fb6).

bboreham commented 7 years ago

Note the fix is in 1.9.8 just released