weaveworks / weave

Simple, resilient multi-host containers networking and more.
https://www.weave.works
Apache License 2.0
6.62k stars 670 forks source link

After 1.5.0 Update DNS Name Resolution Using the Network Plugin Does Not Work #2191

Closed weavebug3256 closed 8 years ago

weavebug3256 commented 8 years ago

I am using the Docker Network Plugin (--net=weave) to communicate between containers that reside on different hosts. Docker + Weave up & running on all hosts.

root@host1:/# weave status
        Version: 1.5.0 (up to date; next check at 2016/04/19 21:31:55)

        Service: router
       Protocol: weave 1..2
           Name: <host name & address>
     Encryption: disabled
  PeerDiscovery: enabled
        Targets: 1
    Connections: 2 (2 established)
          Peers: 3 (with 6 established connections)
 TrustedSubnets: none

        Service: ipam
         Status: ready
          Range: 10.32.0.0-10.47.255.255
  DefaultSubnet: 10.32.0.0/12

        Service: dns
         Domain: weave.local.
       Upstream: <DNS addresses>
            TTL: 1
        Entries: 0

        Service: proxy
        Address: unix:///var/run/weave/weave.sock

        Service: plugin
     DriverName: weave
root@host1:/# docker --version
Docker version 1.11.0, build 4dc5990

My test example is going to be very similar to the instructions in the documentation.

root@host1:/# docker run -it --name a1 --net=weave --hostname a1.weave.local $(weave dns-args) ubuntu:trusty
root@host2:/# docker run -it --name a2 --net=weave --hostname a2.weave.local $(weave dns-args) ubuntu:trusty

At this point weave status dns shows:

root@host1:/# weave status dns
a1           10.32.0.1       <container ID> <MAC address>
a2           10.40.0.1       <container ID> <MAC address>

But when I try to ping one of the containers (from the other container, of course) using their hostname, I get ping: unknown host a1 (or a2). Pinging using the IP address works.

The most strange part is that if I use the Docker API Proxy, it all works as expected. However I don't want to go that route because I automatize all this process and I don't want to think about environment variables, whether I do it all with sudo or in a root shell, whether I am in the same shell when I eval $(weave env) and run my docker container, etc.

root@host1:/# eval $(weave env)
root@host1:/# docker run -it --name a1 ubuntu:trusty
root@host2:/# eval $(weave env)
root@host2:/# docker run -it --name a2 ubuntu:trusty
root@a2:/# ping a1
PING a1.weave.local (10.32.0.1) 56(84) bytes of data.
64 bytes from a1.weave.local (10.32.0.1): icmp_seq=1 ttl=64 time=0.491 ms
64 bytes from a1.weave.local (10.32.0.1): icmp_seq=2 ttl=64 time=0.455 ms
^C
--- a1.weave.local ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.455/0.473/0.491/0.018 ms

Not to mention I didn't have this problem with version 1.4.6. Multiple reinstallations of weave and creating new containers didn't help.

Here is the log of weaveplugin:

root@host1:/# docker logs weaveplugin
INFO: 2016/04/19 19:18:12.253230 Weave plugin 1.5.0 Command line options: [--log-level=debug]
INFO: 2016/04/19 19:18:12.254587 Docker API on unix:///var/run/weave/weave.sock: &[KernelVersion=3.19.0-58-generic BuildTime=2016-04-13T18:34:23.204131695+00:00 Version=1.11.0 ApiVersion=1.23 GitCommit=4dc5990 GoVersion=go1.5.4 Os=linux Arch=amd64]
INFO: 2016/04/19 19:18:12.254736 Listening on /run/docker/plugins/weave.sock for global scope
INFO: 2016/04/19 19:18:12.254841 Listening on /run/docker/plugins/weavemesh.sock for local scope
DEBU: 2016/04/19 19:18:12.264378 [net] ContainerStarted: fe335f5ffe09ff330e68d3bdaa3e2c48cc003fb51803eb34692768461ae0805b
DEBU: 2016/04/19 19:18:12.264413 [net] ContainerStarted: fe335f5ffe09ff330e68d3bdaa3e2c48cc003fb51803eb34692768461ae0805b
INFO: 2016/04/19 19:18:12.333025 [ipam] RequestPool weavelocal   map[]
DEBU: 2016/04/19 19:18:12.333064 weave GET to http://127.0.0.1:6784/ipinfo/defaultsubnet with map[]
DEBU: 2016/04/19 19:18:12.333930 [ipam] RequestPool result weave-10.32.0.0/12-10.32.0.0/12 10.32.0.0/12 map[com.docker.network.gateway:10.32.0.0/12]
DEBU: 2016/04/19 19:18:12.334500 [net] CreateNetwork &{NetworkID:14a5ff7efbc8528c3bbad06863abd2ee8679d5cf555f93cab4e62193e117a0d6 Options:map[com.docker.network.enable_ipv6:false com.docker.network.generic:map[]] IPv4Data:[{AddressSpace: Pool:10.32.0.0/12 Gateway:10.32.0.0/12 AuxAddresses:map[]}] IPv6Data:[]}
INFO: 2016/04/19 19:18:12.334524 [net] CreateNetwork 14a5ff7efbc8528c3bbad06863abd2ee8679d5cf555f93cab4e62193e117a0d6
DEBU: 2016/04/19 19:18:53.266363 [net] ContainerStarted: 40b822726743a654ebe2765ebb549104d2d20ac90a87c17a59fae50a4abe1c65
DEBU: 2016/04/19 19:18:53.266363 [net] ContainerStarted: 40b822726743a654ebe2765ebb549104d2d20ac90a87c17a59fae50a4abe1c65
INFO: 2016/04/19 19:18:53.536062 [ipam] RequestAddress weave-10.32.0.0/12-10.32.0.0/12 <nil> map[]
DEBU: 2016/04/19 19:18:53.536148 weave POST to http://127.0.0.1:6784/ip/_/10.32.0.0/12 with map[]
DEBU: 2016/04/19 19:18:53.545610 [ipam] RequestAddress result 10.32.0.1/12
DEBU: 2016/04/19 19:18:53.546429 [net] CreateEndpoint &{NetworkID:14a5ff7efbc8528c3bbad06863abd2ee8679d5cf555f93cab4e62193e117a0d6 EndpointID:adc859ded150df380cf9a3a897d3050d593410fa45038184fe3aefc66e872932 Interface:0xc820269c20 Options:map[com.docker.network.endpoint.exposedports:[] com.docker.network.portmap:[]]}
INFO: 2016/04/19 19:18:53.546461 [net] CreateEndpoint adc859ded150df380cf9a3a897d3050d593410fa45038184fe3aefc66e872932
DEBU: 2016/04/19 19:18:53.546511 [net] CreateEndpoint &{Response:{Err:} Interface:<nil>}
DEBU: 2016/04/19 19:18:53.573604 [net] JoinEndpoint &{NetworkID:14a5ff7efbc8528c3bbad06863abd2ee8679d5cf555f93cab4e62193e117a0d6 EndpointID:adc859ded150df380cf9a3a897d3050d593410fa45038184fe3aefc66e872932 SandboxKey:/var/run/docker/netns/670a19229df7 Options:map[com.docker.network.endpoint.exposedports:[] com.docker.network.portmap:[]]}
INFO: 2016/04/19 19:18:53.573654 [net] JoinEndpoint 14a5ff7efbc8528c3bbad06863abd2ee8679d5cf555f93cab4e62193e117a0d6:adc859ded150df380cf9a3a897d3050d593410fa45038184fe3aefc66e872932 to /var/run/docker/netns/670a19229df7
DEBU: 2016/04/19 19:18:53.574995 [net] JoinEndpoint &{Response:{Err:} InterfaceName:0xc8202e85d0 Gateway: GatewayIPv6: StaticRoutes:[{Destination:224.0.0.0/4 RouteType:1 NextHop:}] DisableGatewayService:false}
DEBU: 2016/04/19 19:18:53.814220 [net] ContainerStarted: 97cc46f484101b2af749e3de5a34982a3d7b32bf2b3ab62336e1892bfe82ce12
DEBU: 2016/04/19 19:18:53.814220 [net] ContainerStarted: 97cc46f484101b2af749e3de5a34982a3d7b32bf2b3ab62336e1892bfe82ce12
DEBU: 2016/04/19 19:18:53.816522 weave PUT to http://127.0.0.1:6784/name/97cc46f484101b2af749e3de5a34982a3d7b32bf2b3ab62336e1892bfe82ce12/10.32.0.1 with map[fqdn:[a1.weave.local.]]
DEBU: 2016/04/19 19:18:54.156120 [net] ContainerStarted: 8570a92c4da69108435d27ff90d8fa7c620f578e4f04af11818457d2d002406c
DEBU: 2016/04/19 19:18:54.156138 [net] ContainerStarted: 8570a92c4da69108435d27ff90d8fa7c620f578e4f04af11818457d2d002406c
DEBU: 2016/04/19 19:19:49.576965 [net] LeaveEndpoint &{NetworkID:14a5ff7efbc8528c3bbad06863abd2ee8679d5cf555f93cab4e62193e117a0d6 EndpointID:adc859ded150df380cf9a3a897d3050d593410fa45038184fe3aefc66e872932}
INFO: 2016/04/19 19:19:49.577035 [net] LeaveEndpoint 14a5ff7efbc8528c3bbad06863abd2ee8679d5cf555f93cab4e62193e117a0d6:adc859ded150df380cf9a3a897d3050d593410fa45038184fe3aefc66e872932
DEBU: 2016/04/19 19:19:49.733979 [net] DeleteEndpoint &{NetworkID:14a5ff7efbc8528c3bbad06863abd2ee8679d5cf555f93cab4e62193e117a0d6 EndpointID:adc859ded150df380cf9a3a897d3050d593410fa45038184fe3aefc66e872932}
INFO: 2016/04/19 19:19:49.734024 [net] DeleteEndpoint adc859ded150df380cf9a3a897d3050d593410fa45038184fe3aefc66e872932
INFO: 2016/04/19 19:19:49.734572 [ipam] ReleaseAddress weave-10.32.0.0/12-10.32.0.0/12 10.32.0.1
DEBU: 2016/04/19 19:19:49.734619 weave DELETE to http://127.0.0.1:6784/ip/10.32.0.1 with map[]

Thank you.

bboreham commented 8 years ago

I tried to replicate your problem, but it worked for me:

vagrant@host1:~$ weave reset
vagrant@host1:~$ weave launch 192.168.48.12
root@host2:~# weave reset
root@host2:~# weave launch 192.168.48.11
vagrant@host1:~$ docker run -it --name a1 --net=weave --hostname a1.weave.local $(weave dns-args) ubuntu:trusty
root@a1:/# ping a2
PING a2 (10.40.0.0) 56(84) bytes of data.
64 bytes from a2.weave.local (10.40.0.0): icmp_seq=1 ttl=64 time=0.397 ms
64 bytes from a2.weave.local (10.40.0.0): icmp_seq=2 ttl=64 time=0.374 ms
64 bytes from a2.weave.local (10.40.0.0): icmp_seq=3 ttl=64 time=0.378 ms
^C
--- a2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 1998ms
rtt min/avg/max/mdev = 0.374/0.383/0.397/0.010 ms
root@a1:/# ping a2.weave.local
PING a2.weave.local (10.40.0.0) 56(84) bytes of data.
64 bytes from a2.weave.local (10.40.0.0): icmp_seq=1 ttl=64 time=0.399 ms
64 bytes from a2.weave.local (10.40.0.0): icmp_seq=2 ttl=64 time=0.365 ms
^C
--- a2.weave.local ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 999ms
rtt min/avg/max/mdev = 0.365/0.382/0.399/0.017 ms
root@host2:~# docker run -it --name a2 --net=weave --hostname a2.weave.local $(weave dns-args) ubuntu:trusty
root@a2:/# ping a1
PING a1 (10.32.0.1) 56(84) bytes of data.
64 bytes from a1.weave.local (10.32.0.1): icmp_seq=1 ttl=64 time=1.37 ms
64 bytes from a1.weave.local (10.32.0.1): icmp_seq=2 ttl=64 time=0.365 ms

Your weave status dns output indicates all the addresses are known to WeaveDNS.

Could you do docker inspect -f '{{.HostConfig.DNS}}' a1 and confirm that the address there matches the current output of weave dns-args ?

The next step would be to turn on debug logging; address resolution should show up in docker logs weave as something like "lookup a2.weave.local."

To run with debug logging, shut everything down then:

weave launch --log-level=debug
weavebug3256 commented 8 years ago

Yes, those addresses match:

root@host1:/# weave dns-args
--dns 172.17.0.1 --dns-search=weave.local.root
root@host1:/# docker inspect -f '{{.HostConfig.DNS}}' a1
[172.17.0.1]

Here is the output of weave status dns and docker logs weave after weave launch --log-level=debug. But I don't see anything including neither lookup nor a2.weave.local. However I noticed this: [<IP address of host2>:6783] error during connection attempt: dial tcp4 :0-><IP address of host2>:6783: getsockopt: connection refused I don't know what causes this though. All of the communication originating from any of the hosts is allowed on ufw and the hosts can ping each other outside of Docker/Weave.

root@host1:/# weave status dns
a1           10.32.0.1       <container ID> <MAC address>
a2           10.40.0.0       <container ID> <MAC address>
root@host1:/# docker logs weave
INFO: 2016/04/20 14:00:22.499167 Command line options: map[log-level:debug nickname:host1 ipalloc-range:10.32.0.0/12 dns-effective-listen-address:172.17.0.1 dns-listen-address:172.17.0.1:53 http-addr:127.0.0.1:6784 name:<MAC address of host1> port:6783 datapath:datapath]
INFO: 2016/04/20 14:00:22.559774 Communication between peers is unencrypted.
INFO: 2016/04/20 14:00:22.599623 Our name is <MAC address of host1>(host1)
INFO: 2016/04/20 14:00:22.599659 Initial set of peers: [host2]
INFO: 2016/04/20 14:00:22.600977 Docker API on unix:///var/run/docker.sock: &[BuildTime=2016-04-13T18:34:23.204131695+00:00 Version=1.11.0 ApiVersion=1.23 GitCommit=4dc5990 GoVersion=go1.5.4 Os=linux Arch=amd64 KernelVersion=3.19.0-58-generic]
INFO: 2016/04/20 14:00:22.601059 [allocator <MAC address of host1>] Initialising via deferred consensus
INFO: 2016/04/20 14:00:22.602112 Listening for DNS queries on 172.17.0.1
INFO: 2016/04/20 14:00:22.602157 Sniffing traffic on datapath (via ODP)
INFO: 2016/04/20 14:00:22.602647 ->[<IP address of host2>:6783] attempting connection
INFO: 2016/04/20 14:00:22.603103 ->[<IP address of host2>:6783] error during connection attempt: dial tcp4 :0-><IP address of host2>:6783: getsockopt: connection refused
INFO: 2016/04/20 14:00:22.604104 Listening for HTTP control messages on 127.0.0.1:6784
DEBU: 2016/04/20 14:00:22.791856 ODP miss map[15:BlobFlowKey{type: 15, key: 3e010000, mask: ffffffff} 20:BlobFlowKey{type: 20, key: 00000000, mask: ffffffff} 6:BlobFlowKey{type: 6, key: 86dd, mask: ffff} 19:BlobFlowKey{type: 19, key: 00000000, mask: ffffffff} 3:InPortFlowKey{vport: 0} 4:EthernetFlowKey{src: c2:d4:b7:7d:51:4c, dst: 33:33:00:00:00:16} 2:BlobFlowKey{type: 2, key: 07000000, mask: ffffffff} 8:BlobFlowKey{type: 8, key: 00000000000000000000000000000000ff020000000000000000000000000016000000003a000100, mask: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff} 12:BlobFlowKey{type: 12, key: 8f00, mask: ffff}] on port 0
INFO: 2016/04/20 14:00:22.791983 Discovered local MAC c2:d4:b7:7d:51:4c
DEBU: 2016/04/20 14:00:22.792102 Creating ODP flow FlowSpec{keys: [EthernetFlowKey{src: c2:d4:b7:7d:51:4c, dst: 33:33:00:00:00:16} InPortFlowKey{vport: 0}], actions: [OutputAction{vport: 1}]}
DEBU: 2016/04/20 14:00:22.987732 ODP miss map[12:BlobFlowKey{type: 12, key: 8f00, mask: ffff} 19:BlobFlowKey{type: 19, key: 00000000, mask: ffffffff} 2:BlobFlowKey{type: 2, key: 07000000, mask: ffffffff} 15:BlobFlowKey{type: 15, key: 00000000, mask: ffffffff} 8:BlobFlowKey{type: 8, key: fe80000000000000780b43fffe4c1845ff020000000000000000000000000016000000003a000100, mask: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff} 3:InPortFlowKey{vport: 1} 4:EthernetFlowKey{src: 7a:0b:43:4c:18:45, dst: 33:33:00:00:00:16} 6:BlobFlowKey{type: 6, key: 86dd, mask: ffff} 20:BlobFlowKey{type: 20, key: 00000000, mask: ffffffff}] on port 1
INFO: 2016/04/20 14:00:22.987787 Discovered local MAC 7a:0b:43:4c:18:45
DEBU: 2016/04/20 14:00:22.987889 Creating ODP flow FlowSpec{keys: [EthernetFlowKey{src: 7a:0b:43:4c:18:45, dst: 33:33:00:00:00:16} InPortFlowKey{vport: 1}], actions: [OutputAction{vport: 0}]}
DEBU: 2016/04/20 14:00:22.988010 ODP miss map[12:BlobFlowKey{type: 12, key: 8500, mask: ffff} 20:BlobFlowKey{type: 20, key: 00000000, mask: ffffffff} 3:InPortFlowKey{vport: 1} 8:BlobFlowKey{type: 8, key: fe80000000000000780b43fffe4c1845ff020000000000000000000000000002000000003a00ff00, mask: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff} 2:BlobFlowKey{type: 2, key: 00000000, mask: ffffffff} 15:BlobFlowKey{type: 15, key: 00000000, mask: ffffffff} 6:BlobFlowKey{type: 6, key: 86dd, mask: ffff} 19:BlobFlowKey{type: 19, key: 00000000, mask: ffffffff} 4:EthernetFlowKey{src: 7a:0b:43:4c:18:45, dst: 33:33:00:00:00:02}] on port 1
DEBU: 2016/04/20 14:00:22.988092 Creating ODP flow FlowSpec{keys: [EthernetFlowKey{src: 7a:0b:43:4c:18:45, dst: 33:33:00:00:00:02} InPortFlowKey{vport: 1}], actions: [OutputAction{vport: 0}]}
DEBU: 2016/04/20 14:00:22.991601 ODP miss map[6:BlobFlowKey{type: 6, key: 86dd, mask: ffff} 3:InPortFlowKey{vport: 1} 15:BlobFlowKey{type: 15, key: 00000000, mask: ffffffff} 4:EthernetFlowKey{src: <MAC address of host1>, dst: 33:33:00:00:00:16} 20:BlobFlowKey{type: 20, key: 00000000, mask: ffffffff} 19:BlobFlowKey{type: 19, key: 00000000, mask: ffffffff} 8:BlobFlowKey{type: 8, key: fe800000000000003c19acfffea942dbff020000000000000000000000000016000000003a000100, mask: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff} 12:BlobFlowKey{type: 12, key: 8f00, mask: ffff} 2:BlobFlowKey{type: 2, key: 07000000, mask: ffffffff}] on port 1
INFO: 2016/04/20 14:00:22.991627 Discovered local MAC <MAC address of host1>
DEBU: 2016/04/20 14:00:22.991681 Creating ODP flow FlowSpec{keys: [EthernetFlowKey{src: <MAC address of host1>, dst: 33:33:00:00:00:16} InPortFlowKey{vport: 1}], actions: [OutputAction{vport: 0}]}
DEBU: 2016/04/20 14:00:22.991769 ODP miss map[20:BlobFlowKey{type: 20, key: 00000000, mask: ffffffff} 8:BlobFlowKey{type: 8, key: fe800000000000003c19acfffea942dbff020000000000000000000000000002000000003a00ff00, mask: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff} 4:EthernetFlowKey{src: <MAC address of host1>, dst: 33:33:00:00:00:02} 6:BlobFlowKey{type: 6, key: 86dd, mask: ffff} 2:BlobFlowKey{type: 2, key: 00000000, mask: ffffffff} 15:BlobFlowKey{type: 15, key: 00000000, mask: ffffffff} 12:BlobFlowKey{type: 12, key: 8500, mask: ffff} 19:BlobFlowKey{type: 19, key: 00000000, mask: ffffffff} 3:InPortFlowKey{vport: 1}] on port 1
DEBU: 2016/04/20 14:00:22.991826 Creating ODP flow FlowSpec{keys: [EthernetFlowKey{src: <MAC address of host1>, dst: 33:33:00:00:00:02} InPortFlowKey{vport: 1}], actions: [OutputAction{vport: 0}]}
DEBU: 2016/04/20 14:00:22.995613 ODP miss map[20:BlobFlowKey{type: 20, key: 00000000, mask: ffffffff} 3:InPortFlowKey{vport: 0} 8:BlobFlowKey{type: 8, key: fe80000000000000c0d4b7fffe7d514cff020000000000000000000000000002000000003a00ff00, mask: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff} 19:BlobFlowKey{type: 19, key: 00000000, mask: ffffffff} 4:EthernetFlowKey{src: c2:d4:b7:7d:51:4c, dst: 33:33:00:00:00:02} 12:BlobFlowKey{type: 12, key: 8500, mask: ffff} 2:BlobFlowKey{type: 2, key: 00000000, mask: ffffffff} 15:BlobFlowKey{type: 15, key: 00000000, mask: ffffffff} 6:BlobFlowKey{type: 6, key: 86dd, mask: ffff}] on port 0
DEBU: 2016/04/20 14:00:22.995690 Creating ODP flow FlowSpec{keys: [EthernetFlowKey{src: c2:d4:b7:7d:51:4c, dst: 33:33:00:00:00:02} InPortFlowKey{vport: 0}], actions: [OutputAction{vport: 1}]}
INFO: 2016/04/20 14:00:23.984205 ->[<IP address of host2>:6783] attempting connection
INFO: 2016/04/20 14:00:23.984875 ->[<IP address of host2>:6783] error during connection attempt: dial tcp4 :0-><IP address of host2>:6783: getsockopt: connection refused
INFO: 2016/04/20 14:00:27.420411 ->[<IP address of host2>:43463] connection accepted
2016/04/20 14:00:27 ->[<IP address of host2>:43463|<MAC address of host2>(host2)]: connection ready; using protocol version 2
INFO: 2016/04/20 14:00:27.421839 overlay_switch ->[<MAC address of host2>(host2)] using fastdp
2016/04/20 14:00:27 ->[<IP address of host2>:43463|<MAC address of host2>(host2)]: connection added (new peer)
DEBU: 2016/04/20 14:00:27.422138 InvalidateRoutes
DEBU: 2016/04/20 14:00:27.422881 ODP miss map[20:BlobFlowKey{type: 20, key: 00000000, mask: ffffffff} 19:BlobFlowKey{type: 19, key: 00000000, mask: ffffffff} 2:BlobFlowKey{type: 2, key: 00000000, mask: ffffffff} 16:TunnelFlowKey{id: 000000000092b43a, ipv4src: <IP address of host2>, ipv4dst: <IP address of host1>, ttl: 64, tpsrc: 41402, tpdst: 6784} 3:InPortFlowKey{vport: 2} 15:BlobFlowKey{type: 15, key: 00000000, mask: ffffffff} 4:EthernetFlowKey{src: 00:00:00:00:00:00, dst: 00:00:00:00:00:00}] on port 2
DEBU: 2016/04/20 14:00:27.422942 fastdp ->[<nil>|<MAC address of host2>(host2)]: confirmed
DEBU: 2016/04/20 14:00:27.423269 sleeve ->[<nil>|<MAC address of host2>(host2)]: Confirm
DEBU: 2016/04/20 14:00:27.423455 sleeve ->[<nil>|<MAC address of host2>(host2)]: confirmed
DEBU: 2016/04/20 14:00:27.423966 [allocator <MAC address of host1>]: Allocator.OnGossip: 550 bytes
DEBU: 2016/04/20 14:00:27.424801 InvalidateRoutes
DEBU: 2016/04/20 14:00:27.923021 ODP miss map[3:InPortFlowKey{vport: 2} 15:BlobFlowKey{type: 15, key: 00000000, mask: ffffffff} 4:EthernetFlowKey{src: 00:00:00:00:00:00, dst: 00:00:00:00:00:00} 20:BlobFlowKey{type: 20, key: 00000000, mask: ffffffff} 19:BlobFlowKey{type: 19, key: 00000000, mask: ffffffff} 2:BlobFlowKey{type: 2, key: 00000000, mask: ffffffff} 16:TunnelFlowKey{id: 000000000092b43a, ipv4src: <IP address of host2>, ipv4dst: <IP address of host1>, ttl: 64, tpsrc: 41402, tpdst: 6784}] on port 2
DEBU: 2016/04/20 14:00:27.923089 sleeve ->[<nil>|<MAC address of host2>(host2)]: handleHeartbeat
DEBU: 2016/04/20 14:00:27.923109 fastdp ->[<nil>|<MAC address of host2>(host2)]: handleVxlanSpecialPacket
DEBU: 2016/04/20 14:00:27.923170 sleeve ->[<IP address of host2>:6783|<MAC address of host2>(host2)]: sendHeartbeat
DEBU: 2016/04/20 14:00:27.923334 fastdp ->[<IP address of host2>:6784|<MAC address of host2>(host2)]: sendHeartbeat
DEBU: 2016/04/20 14:00:27.924661 InvalidateRoutes
DEBU: 2016/04/20 14:00:27.924802 fastdp ->[<IP address of host2>:6784|<MAC address of host2>(host2)]: handleHeartbeatAck
2016/04/20 14:00:27 ->[<IP address of host2>:43463|<MAC address of host2>(host2)]: connection fully established
DEBU: 2016/04/20 14:00:27.925002 InvalidateRoutes
DEBU: 2016/04/20 14:00:27.925116 sleeve ->[<IP address of host2>:6783|<MAC address of host2>(host2)]: handleHeartbeatAck
DEBU: 2016/04/20 14:00:27.925173 sleeve ->[<IP address of host2>:6783|<MAC address of host2>(host2)]: sendFragTest
INFO: 2016/04/20 14:00:27.925601 EMSGSIZE on send, expecting PMTU update (IP packet was 60028 bytes, payload was 60020 bytes)
DEBU: 2016/04/20 14:00:27.925661 sleeve ->[<IP address of host2>:6783|<MAC address of host2>(host2)]: sendMTUTest: mtu candidate 1438
DEBU: 2016/04/20 14:00:27.926572 sleeve ->[<IP address of host2>:6783|<MAC address of host2>(host2)]: handleFragTestAck
DEBU: 2016/04/20 14:00:27.926852 sleeve ->[<IP address of host2>:6783|<MAC address of host2>(host2)]: handleMTUTestAck: for mtu candidate 1438
DEBU: 2016/04/20 14:00:27.926890 sleeve ->[<IP address of host2>:6783|<MAC address of host2>(host2)]: searchMTU: 1438 1439
INFO: 2016/04/20 14:00:27.926922 sleeve ->[<IP address of host2>:6783|<MAC address of host2>(host2)]: Effective MTU verified at 1438
DEBU: 2016/04/20 14:00:28.031982 ODP miss map[6:BlobFlowKey{type: 6, key: 86dd, mask: ffff} 8:BlobFlowKey{type: 8, key: fe8000000000000090b37bfffea757d9ff020000000000000000000000000016000000003a000100, mask: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff} 2:BlobFlowKey{type: 2, key: 00000000, mask: ffffffff} 3:InPortFlowKey{vport: 2} 4:EthernetFlowKey{src: 92:b3:7b:a7:57:d9, dst: 33:33:00:00:00:16} 15:BlobFlowKey{type: 15, key: 00000000, mask: ffffffff} 19:BlobFlowKey{type: 19, key: 00000000, mask: ffffffff} 16:TunnelFlowKey{id: 000000000092b43a, ipv4src: <IP address of host2>, ipv4dst: <IP address of host1>, ttl: 64, tpsrc: 41141, tpdst: 6784} 20:BlobFlowKey{type: 20, key: 00000000, mask: ffffffff} 12:BlobFlowKey{type: 12, key: 8f00, mask: ffff}] on port 2
INFO: 2016/04/20 14:00:28.032060 Discovered remote MAC 92:b3:7b:a7:57:d9 at <MAC address of host2>(host2)
DEBU: 2016/04/20 14:00:28.032264 Creating ODP flow FlowSpec{keys: [InPortFlowKey{vport: 2} TunnelFlowKey{id: 000000000092b43a, ipv4src: <IP address of host2>, ipv4dst: <IP address of host1>} EthernetFlowKey{src: 92:b3:7b:a7:57:d9, dst: 33:33:00:00:00:16}], actions: [OutputAction{vport: 1} OutputAction{vport: 0}]}
DEBU: 2016/04/20 14:00:28.032458 ODP miss map[20:BlobFlowKey{type: 20, key: 00000000, mask: ffffffff} 12:BlobFlowKey{type: 12, key: 8500, mask: ffff} 15:BlobFlowKey{type: 15, key: 00000000, mask: ffffffff} 6:BlobFlowKey{type: 6, key: 86dd, mask: ffff} 8:BlobFlowKey{type: 8, key: fe8000000000000090b37bfffea757d9ff020000000000000000000000000002000000003a00ff00, mask: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff} 19:BlobFlowKey{type: 19, key: 00000000, mask: ffffffff} 2:BlobFlowKey{type: 2, key: 00000000, mask: ffffffff} 16:TunnelFlowKey{id: 000000000092b43a, ipv4src: <IP address of host2>, ipv4dst: <IP address of host1>, ttl: 64, tpsrc: 57743, tpdst: 6784} 3:InPortFlowKey{vport: 2} 4:EthernetFlowKey{src: 92:b3:7b:a7:57:d9, dst: 33:33:00:00:00:02}] on port 2
DEBU: 2016/04/20 14:00:28.032587 Creating ODP flow FlowSpec{keys: [EthernetFlowKey{src: 92:b3:7b:a7:57:d9, dst: 33:33:00:00:00:02} InPortFlowKey{vport: 2} TunnelFlowKey{id: 000000000092b43a, ipv4src: <IP address of host2>, ipv4dst: <IP address of host1>}], actions: [OutputAction{vport: 1} OutputAction{vport: 0}]}
DEBU: 2016/04/20 14:00:28.032756 ODP miss map[19:BlobFlowKey{type: 19, key: 00000000, mask: ffffffff} 3:InPortFlowKey{vport: 2} 8:BlobFlowKey{type: 8, key: fe8000000000000098f15cfffe10a639ff020000000000000000000000000016000000003a000100, mask: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff} 12:BlobFlowKey{type: 12, key: 8f00, mask: ffff} 20:BlobFlowKey{type: 20, key: 00000000, mask: ffffffff} 2:BlobFlowKey{type: 2, key: 00000000, mask: ffffffff} 16:TunnelFlowKey{id: 000000000092b43a, ipv4src: <IP address of host2>, ipv4dst: <IP address of host1>, ttl: 64, tpsrc: 56435, tpdst: 6784} 4:EthernetFlowKey{src: <MAC address of host2>, dst: 33:33:00:00:00:16} 15:BlobFlowKey{type: 15, key: 00000000, mask: ffffffff} 6:BlobFlowKey{type: 6, key: 86dd, mask: ffff}] on port 2
INFO: 2016/04/20 14:00:28.032792 Discovered remote MAC <MAC address of host2> at <MAC address of host2>(host2)
DEBU: 2016/04/20 14:00:28.032915 Creating ODP flow FlowSpec{keys: [EthernetFlowKey{src: <MAC address of host2>, dst: 33:33:00:00:00:16} InPortFlowKey{vport: 2} TunnelFlowKey{id: 000000000092b43a, ipv4src: <IP address of host2>, ipv4dst: <IP address of host1>}], actions: [OutputAction{vport: 1} OutputAction{vport: 0}]}
DEBU: 2016/04/20 14:00:28.033117 ODP miss map[4:EthernetFlowKey{src: <MAC address of host2>, dst: 33:33:00:00:00:02} 6:BlobFlowKey{type: 6, key: 86dd, mask: ffff} 19:BlobFlowKey{type: 19, key: 00000000, mask: ffffffff} 16:TunnelFlowKey{id: 000000000092b43a, ipv4src: <IP address of host2>, ipv4dst: <IP address of host1>, ttl: 64, tpsrc: 57030, tpdst: 6784} 15:BlobFlowKey{type: 15, key: 00000000, mask: ffffffff} 3:InPortFlowKey{vport: 2} 20:BlobFlowKey{type: 20, key: 00000000, mask: ffffffff} 8:BlobFlowKey{type: 8, key: fe8000000000000098f15cfffe10a639ff020000000000000000000000000002000000003a00ff00, mask: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff} 12:BlobFlowKey{type: 12, key: 8500, mask: ffff} 2:BlobFlowKey{type: 2, key: 00000000, mask: ffffffff}] on port 2
DEBU: 2016/04/20 14:00:28.033248 Creating ODP flow FlowSpec{keys: [TunnelFlowKey{id: 000000000092b43a, ipv4src: <IP address of host2>, ipv4dst: <IP address of host1>} EthernetFlowKey{src: <MAC address of host2>, dst: 33:33:00:00:00:02} InPortFlowKey{vport: 2}], actions: [OutputAction{vport: 1} OutputAction{vport: 0}]}
DEBU: 2016/04/20 14:00:28.363991 ODP miss map[2:BlobFlowKey{type: 2, key: 00000000, mask: ffffffff} 15:BlobFlowKey{type: 15, key: 00000000, mask: ffffffff} 4:EthernetFlowKey{src: fa:2d:72:a5:dd:11, dst: 33:33:00:00:00:16} 8:BlobFlowKey{type: 8, key: fe80000000000000f82d72fffea5dd11ff020000000000000000000000000016000000003a000100, mask: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff} 20:BlobFlowKey{type: 20, key: 00000000, mask: ffffffff} 16:TunnelFlowKey{id: 000000000092b43a, ipv4src: <IP address of host2>, ipv4dst: <IP address of host1>, ttl: 64, tpsrc: 49918, tpdst: 6784} 12:BlobFlowKey{type: 12, key: 8f00, mask: ffff} 3:InPortFlowKey{vport: 2} 6:BlobFlowKey{type: 6, key: 86dd, mask: ffff} 19:BlobFlowKey{type: 19, key: 00000000, mask: ffffffff}] on port 2
INFO: 2016/04/20 14:00:28.364043 Discovered remote MAC fa:2d:72:a5:dd:11 at <MAC address of host2>(host2)
DEBU: 2016/04/20 14:00:28.364206 Creating ODP flow FlowSpec{keys: [TunnelFlowKey{id: 000000000092b43a, ipv4src: <IP address of host2>, ipv4dst: <IP address of host1>} EthernetFlowKey{src: fa:2d:72:a5:dd:11, dst: 33:33:00:00:00:16} InPortFlowKey{vport: 2}], actions: [OutputAction{vport: 1} OutputAction{vport: 0}]}
DEBU: 2016/04/20 14:00:28.364411 ODP miss map[19:BlobFlowKey{type: 19, key: 00000000, mask: ffffffff} 2:BlobFlowKey{type: 2, key: 00000000, mask: ffffffff} 3:InPortFlowKey{vport: 2} 4:EthernetFlowKey{src: fa:2d:72:a5:dd:11, dst: 33:33:00:00:00:02} 6:BlobFlowKey{type: 6, key: 86dd, mask: ffff} 15:BlobFlowKey{type: 15, key: 00000000, mask: ffffffff} 8:BlobFlowKey{type: 8, key: fe80000000000000f82d72fffea5dd11ff020000000000000000000000000002000000003a00ff00, mask: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff} 12:BlobFlowKey{type: 12, key: 8500, mask: ffff} 20:BlobFlowKey{type: 20, key: 00000000, mask: ffffffff} 16:TunnelFlowKey{id: 000000000092b43a, ipv4src: <IP address of host2>, ipv4dst: <IP address of host1>, ttl: 64, tpsrc: 55248, tpdst: 6784}] on port 2
DEBU: 2016/04/20 14:00:28.364540 Creating ODP flow FlowSpec{keys: [TunnelFlowKey{id: 000000000092b43a, ipv4src: <IP address of host2>, ipv4dst: <IP address of host1>} EthernetFlowKey{src: fa:2d:72:a5:dd:11, dst: 33:33:00:00:00:02} InPortFlowKey{vport: 2}], actions: [OutputAction{vport: 1} OutputAction{vport: 0}]}
DEBU: 2016/04/20 14:00:31.003741 ODP miss map[2:BlobFlowKey{type: 2, key: 00000000, mask: ffffffff} 3:InPortFlowKey{vport: 1} 12:BlobFlowKey{type: 12, key: 8500, mask: ffff} 20:BlobFlowKey{type: 20, key: 00000000, mask: ffffffff} 19:BlobFlowKey{type: 19, key: 00000000, mask: ffffffff} 6:BlobFlowKey{type: 6, key: 86dd, mask: ffff} 8:BlobFlowKey{type: 8, key: fe800000000000003c19acfffea942dbff020000000000000000000000000002000000003a00ff00, mask: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff} 15:BlobFlowKey{type: 15, key: 00000000, mask: ffffffff} 4:EthernetFlowKey{src: <MAC address of host1>, dst: 33:33:00:00:00:02}] on port 1
DEBU: 2016/04/20 14:00:31.004030 Creating ODP flow FlowSpec{keys: [EthernetFlowKey{src: <MAC address of host1>, dst: 33:33:00:00:00:02} InPortFlowKey{vport: 1}], actions: [SetTunnelAction{id: 000000000043a92b, ipv4src: <IP address of host1>, ipv4dst: <IP address of host2>, tos: 0, ttl: 64, df: true, csum: false} OutputAction{vport: 2} OutputAction{vport: 0}]}
DEBU: 2016/04/20 14:00:31.004221 ODP miss map[8:BlobFlowKey{type: 8, key: fe80000000000000780b43fffe4c1845ff020000000000000000000000000002000000003a00ff00, mask: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff} 12:BlobFlowKey{type: 12, key: 8500, mask: ffff} 3:InPortFlowKey{vport: 1} 20:BlobFlowKey{type: 20, key: 00000000, mask: ffffffff} 4:EthernetFlowKey{src: 7a:0b:43:4c:18:45, dst: 33:33:00:00:00:02} 6:BlobFlowKey{type: 6, key: 86dd, mask: ffff} 19:BlobFlowKey{type: 19, key: 00000000, mask: ffffffff} 2:BlobFlowKey{type: 2, key: 00000000, mask: ffffffff} 15:BlobFlowKey{type: 15, key: 00000000, mask: ffffffff}] on port 1
DEBU: 2016/04/20 14:00:31.004396 Creating ODP flow FlowSpec{keys: [EthernetFlowKey{src: 7a:0b:43:4c:18:45, dst: 33:33:00:00:00:02} InPortFlowKey{vport: 1}], actions: [SetTunnelAction{id: 000000000043a92b, ipv4src: <IP address of host1>, ipv4dst: <IP address of host2>, tos: 0, ttl: 64, df: true, csum: false} OutputAction{vport: 2} OutputAction{vport: 0}]}
DEBU: 2016/04/20 14:00:31.011695 ODP miss map[19:BlobFlowKey{type: 19, key: 00000000, mask: ffffffff} 8:BlobFlowKey{type: 8, key: fe80000000000000c0d4b7fffe7d514cff020000000000000000000000000002000000003a00ff00, mask: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff} 4:EthernetFlowKey{src: c2:d4:b7:7d:51:4c, dst: 33:33:00:00:00:02} 12:BlobFlowKey{type: 12, key: 8500, mask: ffff} 15:BlobFlowKey{type: 15, key: 00000000, mask: ffffffff} 6:BlobFlowKey{type: 6, key: 86dd, mask: ffff} 20:BlobFlowKey{type: 20, key: 00000000, mask: ffffffff} 2:BlobFlowKey{type: 2, key: 00000000, mask: ffffffff} 3:InPortFlowKey{vport: 0}] on port 0
DEBU: 2016/04/20 14:00:31.012007 Creating ODP flow FlowSpec{keys: [EthernetFlowKey{src: c2:d4:b7:7d:51:4c, dst: 33:33:00:00:00:02} InPortFlowKey{vport: 0}], actions: [SetTunnelAction{id: 000000000043a92b, ipv4src: <IP address of host1>, ipv4dst: <IP address of host2>, tos: 0, ttl: 64, df: true, csum: false} OutputAction{vport: 2} OutputAction{vport: 1}]}
DEBU: 2016/04/20 14:00:37.924288 ODP miss map[15:BlobFlowKey{type: 15, key: 00000000, mask: ffffffff} 4:EthernetFlowKey{src: 00:00:00:00:00:00, dst: 00:00:00:00:00:00} 20:BlobFlowKey{type: 20, key: 00000000, mask: ffffffff} 19:BlobFlowKey{type: 19, key: 00000000, mask: ffffffff} 2:BlobFlowKey{type: 2, key: 00000000, mask: ffffffff} 16:TunnelFlowKey{id: 000000000092b43a, ipv4src: <IP address of host2>, ipv4dst: <IP address of host1>, ttl: 64, tpsrc: 41402, tpdst: 6784} 3:InPortFlowKey{vport: 2}] on port 2
DEBU: 2016/04/20 14:00:37.924339 sleeve ->[<IP address of host2>:6783|<MAC address of host2>(host2)]: handleHeartbeat
DEBU: 2016/04/20 14:00:37.924381 fastdp ->[<IP address of host2>:6784|<MAC address of host2>(host2)]: handleVxlanSpecialPacket
DEBU: 2016/04/20 14:00:37.925066 fastdp ->[<IP address of host2>:6784|<MAC address of host2>(host2)]: sendHeartbeat
DEBU: 2016/04/20 14:00:37.925317 sleeve ->[<IP address of host2>:6783|<MAC address of host2>(host2)]: sendHeartbeat
DEBU: 2016/04/20 14:00:47.924573 ODP miss map[4:EthernetFlowKey{src: 00:00:00:00:00:00, dst: 00:00:00:00:00:00} 20:BlobFlowKey{type: 20, key: 00000000, mask: ffffffff} 19:BlobFlowKey{type: 19, key: 00000000, mask: ffffffff} 2:BlobFlowKey{type: 2, key: 00000000, mask: ffffffff} 16:TunnelFlowKey{id: 000000000092b43a, ipv4src: <IP address of host2>, ipv4dst: <IP address of host1>, ttl: 64, tpsrc: 41402, tpdst: 6784} 3:InPortFlowKey{vport: 2} 15:BlobFlowKey{type: 15, key: 00000000, mask: ffffffff}] on port 2
DEBU: 2016/04/20 14:00:47.924665 fastdp ->[<IP address of host2>:6784|<MAC address of host2>(host2)]: handleVxlanSpecialPacket
DEBU: 2016/04/20 14:00:47.924680 sleeve ->[<IP address of host2>:6783|<MAC address of host2>(host2)]: handleHeartbeat
DEBU: 2016/04/20 14:00:47.925484 fastdp ->[<IP address of host2>:6784|<MAC address of host2>(host2)]: sendHeartbeat
DEBU: 2016/04/20 14:00:47.925484 sleeve ->[<IP address of host2>:6783|<MAC address of host2>(host2)]: sendHeartbeat
INFO: 2016/04/20 14:00:56.565574 Assuming quorum size of 2
DEBU: 2016/04/20 14:00:56.565614 [allocator <MAC address of host1>] Paxos proposing
DEBU: 2016/04/20 14:00:56.567185 [allocator <MAC address of host1>]: OnGossipBroadcast from <MAC address of host2> : 671 bytes
DEBU: 2016/04/20 14:00:56.568919 [allocator <MAC address of host1>]: OnGossipBroadcast from <MAC address of host2> : 784 bytes
DEBU: 2016/04/20 14:00:56.569239 [allocator <MAC address of host1>]: Paxos consensus: [<MAC address of host1> <MAC address of host2>]
DEBU: 2016/04/20 14:00:56.588924 [allocator <MAC address of host1>]: Allocated 10.32.0.1 for 10.32.0.1 in 10.32.0.0/12
INFO: 2016/04/20 14:00:56.866738 [nameserver <MAC address of host1>] adding entry for 949b520b8bea3160bf1bd05ab56d7f444c560d52ede2ab66dfb586022d528531: a1.weave.local. -> 10.32.0.1
DEBU: 2016/04/20 14:00:57.417843 [allocator <MAC address of host1>]: Allocator.OnGossip: 642 bytes
DEBU: 2016/04/20 14:00:57.924919 sleeve ->[<IP address of host2>:6783|<MAC address of host2>(host2)]: handleHeartbeat
DEBU: 2016/04/20 14:00:57.924944 ODP miss map[4:EthernetFlowKey{src: 00:00:00:00:00:00, dst: 00:00:00:00:00:00} 20:BlobFlowKey{type: 20, key: 00000000, mask: ffffffff} 19:BlobFlowKey{type: 19, key: 00000000, mask: ffffffff} 2:BlobFlowKey{type: 2, key: 00000000, mask: ffffffff} 16:TunnelFlowKey{id: 000000000092b43a, ipv4src: <IP address of host2>, ipv4dst: <IP address of host1>, ttl: 64, tpsrc: 41402, tpdst: 6784} 3:InPortFlowKey{vport: 2} 15:BlobFlowKey{type: 15, key: 00000000, mask: ffffffff}] on port 2
DEBU: 2016/04/20 14:00:57.925014 fastdp ->[<IP address of host2>:6784|<MAC address of host2>(host2)]: handleVxlanSpecialPacket
DEBU: 2016/04/20 14:00:57.925803 fastdp ->[<IP address of host2>:6784|<MAC address of host2>(host2)]: sendHeartbeat
DEBU: 2016/04/20 14:00:57.925843 sleeve ->[<IP address of host2>:6783|<MAC address of host2>(host2)]: sendHeartbeat
DEBU: 2016/04/20 14:01:07.925198 sleeve ->[<IP address of host2>:6783|<MAC address of host2>(host2)]: handleHeartbeat
DEBU: 2016/04/20 14:01:07.925288 ODP miss map[16:TunnelFlowKey{id: 000000000092b43a, ipv4src: <IP address of host2>, ipv4dst: <IP address of host1>, ttl: 64, tpsrc: 41402, tpdst: 6784} 3:InPortFlowKey{vport: 2} 15:BlobFlowKey{type: 15, key: 00000000, mask: ffffffff} 4:EthernetFlowKey{src: 00:00:00:00:00:00, dst: 00:00:00:00:00:00} 20:BlobFlowKey{type: 20, key: 00000000, mask: ffffffff} 19:BlobFlowKey{type: 19, key: 00000000, mask: ffffffff} 2:BlobFlowKey{type: 2, key: 00000000, mask: ffffffff}] on port 2
DEBU: 2016/04/20 14:01:07.925380 fastdp ->[<IP address of host2>:6784|<MAC address of host2>(host2)]: handleVxlanSpecialPacket
DEBU: 2016/04/20 14:01:07.926051 sleeve ->[<IP address of host2>:6783|<MAC address of host2>(host2)]: sendHeartbeat
DEBU: 2016/04/20 14:01:07.926073 fastdp ->[<IP address of host2>:6784|<MAC address of host2>(host2)]: sendHeartbeat
DEBU: 2016/04/20 14:01:17.925486 sleeve ->[<IP address of host2>:6783|<MAC address of host2>(host2)]: handleHeartbeat
DEBU: 2016/04/20 14:01:17.925685 ODP miss map[20:BlobFlowKey{type: 20, key: 00000000, mask: ffffffff} 19:BlobFlowKey{type: 19, key: 00000000, mask: ffffffff} 2:BlobFlowKey{type: 2, key: 00000000, mask: ffffffff} 16:TunnelFlowKey{id: 000000000092b43a, ipv4src: <IP address of host2>, ipv4dst: <IP address of host1>, ttl: 64, tpsrc: 41402, tpdst: 6784} 3:InPortFlowKey{vport: 2} 15:BlobFlowKey{type: 15, key: 00000000, mask: ffffffff} 4:EthernetFlowKey{src: 00:00:00:00:00:00, dst: 00:00:00:00:00:00}] on port 2
DEBU: 2016/04/20 14:01:17.925787 fastdp ->[<IP address of host2>:6784|<MAC address of host2>(host2)]: handleVxlanSpecialPacket
DEBU: 2016/04/20 14:01:17.926227 sleeve ->[<IP address of host2>:6783|<MAC address of host2>(host2)]: sendHeartbeat
DEBU: 2016/04/20 14:01:17.926440 fastdp ->[<IP address of host2>:6784|<MAC address of host2>(host2)]: sendHeartbeat
DEBU: 2016/04/20 14:01:27.418083 [allocator <MAC address of host1>]: Allocator.OnGossip: 644 bytes
DEBU: 2016/04/20 14:01:27.925712 sleeve ->[<IP address of host2>:6783|<MAC address of host2>(host2)]: handleHeartbeat
DEBU: 2016/04/20 14:01:27.925846 ODP miss map[15:BlobFlowKey{type: 15, key: 00000000, mask: ffffffff} 4:EthernetFlowKey{src: 00:00:00:00:00:00, dst: 00:00:00:00:00:00} 20:BlobFlowKey{type: 20, key: 00000000, mask: ffffffff} 19:BlobFlowKey{type: 19, key: 00000000, mask: ffffffff} 2:BlobFlowKey{type: 2, key: 00000000, mask: ffffffff} 16:TunnelFlowKey{id: 000000000092b43a, ipv4src: <IP address of host2>, ipv4dst: <IP address of host1>, ttl: 64, tpsrc: 41402, tpdst: 6784} 3:InPortFlowKey{vport: 2}] on port 2
DEBU: 2016/04/20 14:01:27.925948 fastdp ->[<IP address of host2>:6784|<MAC address of host2>(host2)]: handleVxlanSpecialPacket
DEBU: 2016/04/20 14:01:27.926397 sleeve ->[<IP address of host2>:6783|<MAC address of host2>(host2)]: sendHeartbeat
DEBU: 2016/04/20 14:01:27.926746 fastdp ->[<IP address of host2>:6784|<MAC address of host2>(host2)]: sendHeartbeat
DEBU: 2016/04/20 14:01:37.926063 sleeve ->[<IP address of host2>:6783|<MAC address of host2>(host2)]: handleHeartbeat
DEBU: 2016/04/20 14:01:37.926297 ODP miss map[2:BlobFlowKey{type: 2, key: 00000000, mask: ffffffff} 16:TunnelFlowKey{id: 000000000092b43a, ipv4src: <IP address of host2>, ipv4dst: <IP address of host1>, ttl: 64, tpsrc: 41402, tpdst: 6784} 3:InPortFlowKey{vport: 2} 15:BlobFlowKey{type: 15, key: 00000000, mask: ffffffff} 4:EthernetFlowKey{src: 00:00:00:00:00:00, dst: 00:00:00:00:00:00} 20:BlobFlowKey{type: 20, key: 00000000, mask: ffffffff} 19:BlobFlowKey{type: 19, key: 00000000, mask: ffffffff}] on port 2
DEBU: 2016/04/20 14:01:37.926368 fastdp ->[<IP address of host2>:6784|<MAC address of host2>(host2)]: handleVxlanSpecialPacket
DEBU: 2016/04/20 14:01:37.926711 sleeve ->[<IP address of host2>:6783|<MAC address of host2>(host2)]: sendHeartbeat
DEBU: 2016/04/20 14:01:37.927084 fastdp ->[<IP address of host2>:6784|<MAC address of host2>(host2)]: sendHeartbeat
DEBU: 2016/04/20 14:01:47.926312 sleeve ->[<IP address of host2>:6783|<MAC address of host2>(host2)]: handleHeartbeat
DEBU: 2016/04/20 14:01:47.926544 ODP miss map[20:BlobFlowKey{type: 20, key: 00000000, mask: ffffffff} 19:BlobFlowKey{type: 19, key: 00000000, mask: ffffffff} 2:BlobFlowKey{type: 2, key: 00000000, mask: ffffffff} 16:TunnelFlowKey{id: 000000000092b43a, ipv4src: <IP address of host2>, ipv4dst: <IP address of host1>, ttl: 64, tpsrc: 41402, tpdst: 6784} 3:InPortFlowKey{vport: 2} 15:BlobFlowKey{type: 15, key: 00000000, mask: ffffffff} 4:EthernetFlowKey{src: 00:00:00:00:00:00, dst: 00:00:00:00:00:00}] on port 2
DEBU: 2016/04/20 14:01:47.926642 fastdp ->[<IP address of host2>:6784|<MAC address of host2>(host2)]: handleVxlanSpecialPacket
DEBU: 2016/04/20 14:01:47.926917 sleeve ->[<IP address of host2>:6783|<MAC address of host2>(host2)]: sendHeartbeat
DEBU: 2016/04/20 14:01:47.927455 fastdp ->[<IP address of host2>:6784|<MAC address of host2>(host2)]: sendHeartbeat
DEBU: 2016/04/20 14:01:57.417743 [allocator <MAC address of host1>]: Allocator.OnGossip: 644 bytes
DEBU: 2016/04/20 14:01:57.926612 sleeve ->[<IP address of host2>:6783|<MAC address of host2>(host2)]: handleHeartbeat
DEBU: 2016/04/20 14:01:57.926908 ODP miss map[19:BlobFlowKey{type: 19, key: 00000000, mask: ffffffff} 2:BlobFlowKey{type: 2, key: 00000000, mask: ffffffff} 16:TunnelFlowKey{id: 000000000092b43a, ipv4src: <IP address of host2>, ipv4dst: <IP address of host1>, ttl: 64, tpsrc: 41402, tpdst: 6784} 3:InPortFlowKey{vport: 2} 15:BlobFlowKey{type: 15, key: 00000000, mask: ffffffff} 4:EthernetFlowKey{src: 00:00:00:00:00:00, dst: 00:00:00:00:00:00} 20:BlobFlowKey{type: 20, key: 00000000, mask: ffffffff}] on port 2
DEBU: 2016/04/20 14:01:57.927003 fastdp ->[<IP address of host2>:6784|<MAC address of host2>(host2)]: handleVxlanSpecialPacket
DEBU: 2016/04/20 14:01:57.927166 sleeve ->[<IP address of host2>:6783|<MAC address of host2>(host2)]: sendHeartbeat
DEBU: 2016/04/20 14:01:57.927771 fastdp ->[<IP address of host2>:6784|<MAC address of host2>(host2)]: sendHeartbeat
DEBU: 2016/04/20 14:02:07.926856 sleeve ->[<IP address of host2>:6783|<MAC address of host2>(host2)]: handleHeartbeat
DEBU: 2016/04/20 14:02:07.927085 ODP miss map[15:BlobFlowKey{type: 15, key: 00000000, mask: ffffffff} 4:EthernetFlowKey{src: 00:00:00:00:00:00, dst: 00:00:00:00:00:00} 20:BlobFlowKey{type: 20, key: 00000000, mask: ffffffff} 19:BlobFlowKey{type: 19, key: 00000000, mask: ffffffff} 2:BlobFlowKey{type: 2, key: 00000000, mask: ffffffff} 16:TunnelFlowKey{id: 000000000092b43a, ipv4src: <IP address of host2>, ipv4dst: <IP address of host1>, ttl: 64, tpsrc: 41402, tpdst: 6784} 3:InPortFlowKey{vport: 2}] on port 2
DEBU: 2016/04/20 14:02:07.927161 fastdp ->[<IP address of host2>:6784|<MAC address of host2>(host2)]: handleVxlanSpecialPacket
DEBU: 2016/04/20 14:02:07.927359 sleeve ->[<IP address of host2>:6783|<MAC address of host2>(host2)]: sendHeartbeat
DEBU: 2016/04/20 14:02:07.928032 fastdp ->[<IP address of host2>:6784|<MAC address of host2>(host2)]: sendHeartbeat
INFO: 2016/04/20 14:02:15.920669 [nameserver <MAC address of host1>] container 949b520b8bea3160bf1bd05ab56d7f444c560d52ede2ab66dfb586022d528531 died; tombstoning entry a1.weave.local. -> 10.32.0.1
DEBU: 2016/04/20 14:02:17.927209 sleeve ->[<IP address of host2>:6783|<MAC address of host2>(host2)]: handleHeartbeat
DEBU: 2016/04/20 14:02:17.927440 ODP miss map[2:BlobFlowKey{type: 2, key: 00000000, mask: ffffffff} 16:TunnelFlowKey{id: 000000000092b43a, ipv4src: <IP address of host2>, ipv4dst: <IP address of host1>, ttl: 64, tpsrc: 41402, tpdst: 6784} 3:InPortFlowKey{vport: 2} 15:BlobFlowKey{type: 15, key: 00000000, mask: ffffffff} 4:EthernetFlowKey{src: 00:00:00:00:00:00, dst: 00:00:00:00:00:00} 20:BlobFlowKey{type: 20, key: 00000000, mask: ffffffff} 19:BlobFlowKey{type: 19, key: 00000000, mask: ffffffff}] on port 2
DEBU: 2016/04/20 14:02:17.927550 fastdp ->[<IP address of host2>:6784|<MAC address of host2>(host2)]: handleVxlanSpecialPacket
DEBU: 2016/04/20 14:02:17.927569 sleeve ->[<IP address of host2>:6783|<MAC address of host2>(host2)]: sendHeartbeat
DEBU: 2016/04/20 14:02:17.928299 fastdp ->[<IP address of host2>:6784|<MAC address of host2>(host2)]: sendHeartbeat
DEBU: 2016/04/20 14:02:27.418433 [allocator <MAC address of host1>]: Allocator.OnGossip: 644 bytes
bboreham commented 8 years ago

Thanks for the logs. The 'connection refused' could be that host2 is not up yet - it connects the other way round 5 seconds later.

When using the plugin, DNS lookups should go container -> [Docker DNS on 127.0.0.11] -> WeaveDNS. Evidently they are not arriving at the third step.

Is there anything relevant in the Docker logs? E.g. journalctl -u docker.service

Can you install dig (or use an image that includes it; e.g. aanand/docker-dnsutils) and see what dig a2 gives you? That should show which DNS server is answering.

What is in /etc/resolv.conf inside the container you are pinging from?

weavebug3256 commented 8 years ago

I just realized this while trying to install dnsutils in the container: when I start a container using --net=weave, there is no internet connection at all:

root@host1:/# docker run -it --name a1 --net=weave --hostname a1.weave.local $(weave dns-args) ubuntu:trusty
root@a1:/# apt-get update
Err http://archive.ubuntu.com trusty InRelease

Err http://archive.ubuntu.com trusty-updates InRelease

Err http://archive.ubuntu.com trusty-security InRelease

Err http://archive.ubuntu.com trusty Release.gpg
  Could not resolve 'archive.ubuntu.com'
Err http://archive.ubuntu.com trusty-updates Release.gpg
  Could not resolve 'archive.ubuntu.com'
Err http://archive.ubuntu.com trusty-security Release.gpg
  Could not resolve 'archive.ubuntu.com'
Reading package lists... Done
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty/InRelease  

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/InRelease  

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-security/InRelease  

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty/Release.gpg  Could not resolve 'archive.ubuntu.com'

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-updates/Release.gpg  Could not resolve 'archive.ubuntu.com'

W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/trusty-security/Release.gpg  Could not resolve 'archive.ubuntu.com'

W: Some index files failed to download. They have been ignored, or old ones used instead.
root@a1:/# ping google.com
ping: unknown host google.com

And there is some relevant information in the Docker log, but I don't understand why something like Unable to connect to plugin: <whichever Weave plugin> would occur in the first place.

(The reason that all the messages like No such container: weave have appeared is that while I've been uninstalling and reinstalling Weave x times, sometimes I've typed the wrong commands (e.g. docker rmi weave instead of docker rmi weave:1.5.0), so I guess they can be ignored.)

root@host1:/# cat /var/log/upstart/docker.log
time="2016-04-20T15:28:36.989835344+02:00" level=error msg="Handler for POST /v1.23/containers/9a8b5d672c6c/stop returned error: Container 9a8b5d672c6c is already stopped" 
time="2016-04-20T15:28:37.034920310+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T15:28:37.158292323+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T15:28:37.227278389+02:00" level=error msg="Handler for POST /v1.23/containers/771d99c3b050/stop returned error: Container 771d99c3b050 is already stopped" 
time="2016-04-20T15:28:37.303504907+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T15:28:37.367223829+02:00" level=error msg="Handler for POST /v1.23/containers/2f7a63bd4b34/stop returned error: Container 2f7a63bd4b34 is already stopped" 
time="2016-04-20T15:30:15.079257920+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T15:30:15.337497209+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/NetworkDriver.DeleteNetwork, retrying in 1s" 
time="2016-04-20T15:30:16.337755983+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/NetworkDriver.DeleteNetwork, retrying in 2s" 
time="2016-04-20T15:30:18.338026138+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/NetworkDriver.DeleteNetwork, retrying in 4s" 
time="2016-04-20T15:30:22.338486704+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/NetworkDriver.DeleteNetwork, retrying in 8s" 
time="2016-04-20T15:30:30.338914812+02:00" level=warning msg="driver error deleting network weave : Post http://%2Frun%2Fdocker%2Fplugins%2Fweavemesh.sock/NetworkDriver.DeleteNetwork: dial unix /run/docker/plugins/weavemesh.sock: connect: no such file or directory" 
time="2016-04-20T15:30:30.339172962+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/IpamDriver.ReleaseAddress, retrying in 1s" 
time="2016-04-20T15:30:31.339563790+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/IpamDriver.ReleaseAddress, retrying in 2s" 
time="2016-04-20T15:30:33.339991456+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/IpamDriver.ReleaseAddress, retrying in 4s" 
time="2016-04-20T15:30:37.340390371+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/IpamDriver.ReleaseAddress, retrying in 8s" 
time="2016-04-20T15:30:42.239995230+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T15:30:45.340863363+02:00" level=warning msg="Failed to release gateway ip address 10.32.0.0 on delete of network weave (14a5ff7efbc8528c3bbad06863abd2ee8679d5cf555f93cab4e62193e117a0d6): Post http://%2Frun%2Fdocker%2Fplugins%2Fweavemesh.sock/IpamDriver.ReleaseAddress: dial unix /run/docker/plugins/weavemesh.sock: connect: no such file or directory" 
time="2016-04-20T15:30:45.341093837+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/IpamDriver.ReleasePool, retrying in 1s" 
time="2016-04-20T15:30:46.341530717+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/IpamDriver.ReleasePool, retrying in 2s" 
time="2016-04-20T15:30:48.341912107+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/IpamDriver.ReleasePool, retrying in 4s" 
time="2016-04-20T15:30:52.342382736+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/IpamDriver.ReleasePool, retrying in 8s" 
time="2016-04-20T15:31:00.342844030+02:00" level=warning msg="Failed to release address pool weave-10.32.0.0/12-10.32.0.0/12 on delete of network weave (14a5ff7efbc8528c3bbad06863abd2ee8679d5cf555f93cab4e62193e117a0d6): Post http://%2Frun%2Fdocker%2Fplugins%2Fweavemesh.sock/IpamDriver.ReleasePool: dial unix /run/docker/plugins/weavemesh.sock: connect: no such file or directory" 
time="2016-04-20T15:31:15.516588213+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T15:31:15.764278473+02:00" level=error msg="Handler for GET /v1.18/containers/weavedb/json returned error: No such container: weavedb" 
time="2016-04-20T15:31:15.765007784+02:00" level=error msg="Handler for GET /v1.18/images/weavedb/json returned error: No such image: weavedb" 
time="2016-04-20T15:31:15.893250139+02:00" level=error msg="Handler for GET /v1.18/containers/weaveworks/weave:1.5.0/json returned error: No such container: weaveworks/weave:1.5.0" 
time="2016-04-20T15:31:15.917492118+02:00" level=error msg="Handler for GET /v1.18/containers/weave/json returned error: No such container: weave" 
time="2016-04-20T15:31:15.917993655+02:00" level=error msg="Handler for GET /v1.18/images/weave/json returned error: No such image: weave" 
time="2016-04-20T15:31:16.576012685+02:00" level=error msg="Handler for GET /v1.18/containers/weavevolumes-1.5.0/json returned error: No such container: weavevolumes-1.5.0" 
time="2016-04-20T15:31:16.576497917+02:00" level=error msg="Handler for GET /v1.18/images/weavevolumes-1.5.0/json returned error: No such image: weavevolumes-1.5.0" 
time="2016-04-20T15:31:16.822135958+02:00" level=error msg="Handler for GET /v1.18/containers/weaveworks/weaveexec:1.5.0/json returned error: No such container: weaveworks/weaveexec:1.5.0" 
time="2016-04-20T15:31:16.847395169+02:00" level=error msg="Handler for GET /v1.18/containers/weaveproxy/json returned error: No such container: weaveproxy" 
time="2016-04-20T15:31:16.847966926+02:00" level=error msg="Handler for GET /v1.18/images/weaveproxy/json returned error: No such image: weaveproxy" 
time="2016-04-20T15:31:16.873475611+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T15:31:17.204579404+02:00" level=error msg="Handler for GET /v1.18/containers/weaveworks/plugin:1.5.0/json returned error: No such container: weaveworks/plugin:1.5.0" 
time="2016-04-20T15:31:17.223630779+02:00" level=error msg="Handler for GET /v1.18/containers/weaveplugin/json returned error: No such container: weaveplugin" 
time="2016-04-20T15:31:17.224082366+02:00" level=error msg="Handler for GET /v1.18/images/weaveplugin/json returned error: No such image: weaveplugin" 
time="2016-04-20T15:31:30.826079522+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T15:31:31.094026607+02:00" level=warning msg="Failed to release gateway ip address 10.32.0.0 on delete of network weave (a63fba9848d64a80a329bc14d441bd2c07ab60f0bc48f1915969844e680deeea): IpamDriver.ReleaseAddress: 400 Bad Request: Delete: no addresses for 10.32.0.0\n" 
time="2016-04-20T15:31:31.191727762+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T15:31:31.343319195+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T15:31:31.558026488+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T15:32:00.551631196+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T15:32:51.745057378+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T15:32:51.999991377+02:00" level=error msg="Handler for DELETE /v1.21/networks/weave returned error: network weave not found" 
time="2016-04-20T15:32:52.019369954+02:00" level=error msg="Error setting up exec command in container weaveproxy: Container 0cd2390e9566db296dc4cf9ae4d86132aef91a0d4002371d284bda3a10b4272d is not running" 
time="2016-04-20T15:32:52.019412333+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/exec returned error: Container 0cd2390e9566db296dc4cf9ae4d86132aef91a0d4002371d284bda3a10b4272d is not running" 
time="2016-04-20T15:32:52.061438752+02:00" level=error msg="Handler for POST /v1.18/containers/weaveplugin/stop returned error: Container weaveplugin is already stopped" 
time="2016-04-20T15:32:52.100714718+02:00" level=error msg="Handler for POST /v1.18/containers/weave/stop returned error: Container weave is already stopped" 
time="2016-04-20T15:32:52.136077015+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/stop returned error: Container weaveproxy is already stopped" 
time="2016-04-20T15:33:23.705050582+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T15:33:24.010237167+02:00" level=error msg="Handler for GET /v1.18/containers/weavedb/json returned error: No such container: weavedb" 
time="2016-04-20T15:33:24.010761978+02:00" level=error msg="Handler for GET /v1.18/images/weavedb/json returned error: No such image: weavedb" 
time="2016-04-20T15:33:24.138081253+02:00" level=error msg="Handler for GET /v1.18/containers/weaveworks/weave:1.5.0/json returned error: No such container: weaveworks/weave:1.5.0" 
time="2016-04-20T15:33:24.163161979+02:00" level=error msg="Handler for GET /v1.18/containers/weave/json returned error: No such container: weave" 
time="2016-04-20T15:33:24.163701294+02:00" level=error msg="Handler for GET /v1.18/images/weave/json returned error: No such image: weave" 
time="2016-04-20T15:33:24.819869850+02:00" level=error msg="Handler for GET /v1.18/containers/weavevolumes-1.5.0/json returned error: No such container: weavevolumes-1.5.0" 
time="2016-04-20T15:33:24.820329423+02:00" level=error msg="Handler for GET /v1.18/images/weavevolumes-1.5.0/json returned error: No such image: weavevolumes-1.5.0" 
time="2016-04-20T15:33:25.067407217+02:00" level=error msg="Handler for GET /v1.18/containers/weaveworks/weaveexec:1.5.0/json returned error: No such container: weaveworks/weaveexec:1.5.0" 
time="2016-04-20T15:33:25.086603886+02:00" level=error msg="Handler for GET /v1.18/containers/weaveproxy/json returned error: No such container: weaveproxy" 
time="2016-04-20T15:33:25.086956143+02:00" level=error msg="Handler for GET /v1.18/images/weaveproxy/json returned error: No such image: weaveproxy" 
time="2016-04-20T15:33:25.105861005+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T15:33:25.395198041+02:00" level=error msg="Handler for GET /v1.18/containers/weaveworks/plugin:1.5.0/json returned error: No such container: weaveworks/plugin:1.5.0" 
time="2016-04-20T15:33:25.414475675+02:00" level=error msg="Handler for GET /v1.18/containers/weaveplugin/json returned error: No such container: weaveplugin" 
time="2016-04-20T15:33:25.414885764+02:00" level=error msg="Handler for GET /v1.18/images/weaveplugin/json returned error: No such image: weaveplugin" 
time="2016-04-20T15:33:47.957849981+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T15:34:36.723091154+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T15:34:36.998878348+02:00" level=warning msg="Failed to release gateway ip address 10.32.0.0 on delete of network weave (98aff9de62b1fa6be7ee13a03225afc97cb19628bbc15683272c63f05cef4331): IpamDriver.ReleaseAddress: 400 Bad Request: Delete: no addresses for 10.32.0.0\n" 
time="2016-04-20T15:34:37.288488689+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T15:34:37.460345355+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T15:34:37.626991750+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T15:34:55.785423465+02:00" level=error msg="Handler for POST /v1.23/containers/b55/stop returned error: Container b55 is already stopped" 
time="2016-04-20T15:35:04.036894092+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T15:35:04.288460952+02:00" level=error msg="Handler for DELETE /v1.21/networks/weave returned error: network weave not found" 
time="2016-04-20T15:35:04.306350308+02:00" level=error msg="Error setting up exec command in container weaveproxy: No such container: weaveproxy" 
time="2016-04-20T15:35:04.306386595+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/exec returned error: No such container: weaveproxy" 
time="2016-04-20T15:35:04.330252334+02:00" level=error msg="Handler for GET /v1.18/containers/weave/json returned error: No such container: weave" 
time="2016-04-20T15:35:04.330659965+02:00" level=error msg="Handler for GET /v1.18/images/weave/json returned error: No such image: weave" 
time="2016-04-20T15:35:04.348039620+02:00" level=error msg="Handler for POST /v1.18/containers/weaveplugin/stop returned error: No such container: weaveplugin" 
time="2016-04-20T15:35:04.365064708+02:00" level=error msg="Handler for DELETE /v1.18/containers/weaveplugin returned error: No such container: weaveplugin" 
time="2016-04-20T15:35:04.381866412+02:00" level=error msg="Handler for POST /v1.18/containers/weave/stop returned error: No such container: weave" 
time="2016-04-20T15:35:04.398667245+02:00" level=error msg="Handler for DELETE /v1.18/containers/weave returned error: No such container: weave" 
time="2016-04-20T15:35:04.415710977+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/stop returned error: No such container: weaveproxy" 
time="2016-04-20T15:35:04.433056149+02:00" level=error msg="Handler for DELETE /v1.18/containers/weaveproxy returned error: No such container: weaveproxy" 
time="2016-04-20T15:35:19.083593134+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T15:35:19.345756866+02:00" level=error msg="Handler for DELETE /v1.21/networks/weave returned error: network weave not found" 
time="2016-04-20T15:35:19.363266593+02:00" level=error msg="Error setting up exec command in container weaveproxy: No such container: weaveproxy" 
time="2016-04-20T15:35:19.363300813+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/exec returned error: No such container: weaveproxy" 
time="2016-04-20T15:35:19.386419965+02:00" level=error msg="Handler for GET /v1.18/containers/weave/json returned error: No such container: weave" 
time="2016-04-20T15:35:19.386840098+02:00" level=error msg="Handler for GET /v1.18/images/weave/json returned error: No such image: weave" 
time="2016-04-20T15:35:19.404348539+02:00" level=error msg="Handler for POST /v1.18/containers/weaveplugin/stop returned error: No such container: weaveplugin" 
time="2016-04-20T15:35:19.421324219+02:00" level=error msg="Handler for DELETE /v1.18/containers/weaveplugin returned error: No such container: weaveplugin" 
time="2016-04-20T15:35:19.437957021+02:00" level=error msg="Handler for POST /v1.18/containers/weave/stop returned error: No such container: weave" 
time="2016-04-20T15:35:19.454703665+02:00" level=error msg="Handler for DELETE /v1.18/containers/weave returned error: No such container: weave" 
time="2016-04-20T15:35:19.471414383+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/stop returned error: No such container: weaveproxy" 
time="2016-04-20T15:35:19.488186549+02:00" level=error msg="Handler for DELETE /v1.18/containers/weaveproxy returned error: No such container: weaveproxy" 
time="2016-04-20T15:35:33.436767953+02:00" level=error msg="Handler for POST /v1.23/containers/cc7/stop returned error: No such container: cc7" 
time="2016-04-20T15:35:37.139151799+02:00" level=error msg="Handler for DELETE /v1.23/containers/cc7 returned error: No such container: cc7" 
time="2016-04-20T15:36:25.500728439+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T15:36:25.776766375+02:00" level=error msg="Handler for DELETE /v1.21/networks/weave returned error: network weave not found" 
time="2016-04-20T15:36:25.794953626+02:00" level=error msg="Handler for POST /v1.18/containers/weaveplugin/stop returned error: No such container: weaveplugin" 
time="2016-04-20T15:36:25.812094941+02:00" level=error msg="Handler for POST /v1.18/containers/weave/stop returned error: No such container: weave" 
time="2016-04-20T15:36:25.838674586+02:00" level=error msg="Error setting up exec command in container weaveproxy: No such container: weaveproxy" 
time="2016-04-20T15:36:25.838711761+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/exec returned error: No such container: weaveproxy" 
time="2016-04-20T15:36:25.861168248+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/stop returned error: No such container: weaveproxy" 
time="2016-04-20T15:36:30.503751965+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T15:36:30.768253019+02:00" level=error msg="Handler for DELETE /v1.21/networks/weave returned error: network weave not found" 
time="2016-04-20T15:36:30.786505193+02:00" level=error msg="Error setting up exec command in container weaveproxy: No such container: weaveproxy" 
time="2016-04-20T15:36:30.786542202+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/exec returned error: No such container: weaveproxy" 
time="2016-04-20T15:36:30.809875442+02:00" level=error msg="Handler for GET /v1.18/containers/weave/json returned error: No such container: weave" 
time="2016-04-20T15:36:30.810311128+02:00" level=error msg="Handler for GET /v1.18/images/weave/json returned error: No such image: weave" 
time="2016-04-20T15:36:30.828509361+02:00" level=error msg="Handler for POST /v1.18/containers/weaveplugin/stop returned error: No such container: weaveplugin" 
time="2016-04-20T15:36:30.845266949+02:00" level=error msg="Handler for DELETE /v1.18/containers/weaveplugin returned error: No such container: weaveplugin" 
time="2016-04-20T15:36:30.862182567+02:00" level=error msg="Handler for POST /v1.18/containers/weave/stop returned error: No such container: weave" 
time="2016-04-20T15:36:30.879088527+02:00" level=error msg="Handler for DELETE /v1.18/containers/weave returned error: No such container: weave" 
time="2016-04-20T15:36:30.896339933+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/stop returned error: No such container: weaveproxy" 
time="2016-04-20T15:36:30.913147453+02:00" level=error msg="Handler for DELETE /v1.18/containers/weaveproxy returned error: No such container: weaveproxy" 
time="2016-04-20T15:36:56.149393936+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T15:36:56.407378376+02:00" level=error msg="Handler for DELETE /v1.21/networks/weave returned error: network weave not found" 
time="2016-04-20T15:36:56.425152121+02:00" level=error msg="Error setting up exec command in container weaveproxy: No such container: weaveproxy" 
time="2016-04-20T15:36:56.425191239+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/exec returned error: No such container: weaveproxy" 
time="2016-04-20T15:36:56.448326281+02:00" level=error msg="Handler for GET /v1.18/containers/weave/json returned error: No such container: weave" 
time="2016-04-20T15:36:56.448743468+02:00" level=error msg="Handler for GET /v1.18/images/weave/json returned error: No such image: weave" 
time="2016-04-20T15:36:56.466005756+02:00" level=error msg="Handler for POST /v1.18/containers/weaveplugin/stop returned error: No such container: weaveplugin" 
time="2016-04-20T15:36:56.482561909+02:00" level=error msg="Handler for DELETE /v1.18/containers/weaveplugin returned error: No such container: weaveplugin" 
time="2016-04-20T15:36:56.499184834+02:00" level=error msg="Handler for POST /v1.18/containers/weave/stop returned error: No such container: weave" 
time="2016-04-20T15:36:56.515850689+02:00" level=error msg="Handler for DELETE /v1.18/containers/weave returned error: No such container: weave" 
time="2016-04-20T15:36:56.533047838+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/stop returned error: No such container: weaveproxy" 
time="2016-04-20T15:36:56.549631253+02:00" level=error msg="Handler for DELETE /v1.18/containers/weaveproxy returned error: No such container: weaveproxy" 
time="2016-04-20T15:38:22.068960680+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T15:38:22.318840666+02:00" level=error msg="Handler for DELETE /v1.21/networks/weave returned error: network weave not found" 
time="2016-04-20T15:38:22.339931644+02:00" level=error msg="Handler for POST /v1.18/containers/weaveplugin/stop returned error: No such container: weaveplugin" 
time="2016-04-20T15:38:22.358673356+02:00" level=error msg="Handler for POST /v1.18/containers/weave/stop returned error: No such container: weave" 
time="2016-04-20T15:38:22.378883997+02:00" level=error msg="Error setting up exec command in container weaveproxy: No such container: weaveproxy" 
time="2016-04-20T15:38:22.378934115+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/exec returned error: No such container: weaveproxy" 
time="2016-04-20T15:38:22.397117215+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/stop returned error: No such container: weaveproxy" 
time="2016-04-20T15:38:27.388130947+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T15:38:27.639337049+02:00" level=error msg="Handler for DELETE /v1.21/networks/weave returned error: network weave not found" 
time="2016-04-20T15:38:27.657229183+02:00" level=error msg="Error setting up exec command in container weaveproxy: No such container: weaveproxy" 
time="2016-04-20T15:38:27.657266341+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/exec returned error: No such container: weaveproxy" 
time="2016-04-20T15:38:27.680746627+02:00" level=error msg="Handler for GET /v1.18/containers/weave/json returned error: No such container: weave" 
time="2016-04-20T15:38:27.681156218+02:00" level=error msg="Handler for GET /v1.18/images/weave/json returned error: No such image: weave" 
time="2016-04-20T15:38:27.698589035+02:00" level=error msg="Handler for POST /v1.18/containers/weaveplugin/stop returned error: No such container: weaveplugin" 
time="2016-04-20T15:38:27.715600677+02:00" level=error msg="Handler for DELETE /v1.18/containers/weaveplugin returned error: No such container: weaveplugin" 
time="2016-04-20T15:38:27.732347180+02:00" level=error msg="Handler for POST /v1.18/containers/weave/stop returned error: No such container: weave" 
time="2016-04-20T15:38:27.749017675+02:00" level=error msg="Handler for DELETE /v1.18/containers/weave returned error: No such container: weave" 
time="2016-04-20T15:38:27.766165112+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/stop returned error: No such container: weaveproxy" 
time="2016-04-20T15:38:27.783424286+02:00" level=error msg="Handler for DELETE /v1.18/containers/weaveproxy returned error: No such container: weaveproxy" 
time="2016-04-20T15:39:20.084157563+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T15:39:20.385115867+02:00" level=error msg="Handler for GET /v1.18/containers/weavedb/json returned error: No such container: weavedb" 
time="2016-04-20T15:39:20.385674182+02:00" level=error msg="Handler for GET /v1.18/images/weavedb/json returned error: No such image: weavedb" 
time="2016-04-20T15:39:20.514816085+02:00" level=error msg="Handler for GET /v1.18/containers/weaveworks/weave:1.5.0/json returned error: No such container: weaveworks/weave:1.5.0" 
time="2016-04-20T15:39:20.534402084+02:00" level=error msg="Handler for GET /v1.18/containers/weave/json returned error: No such container: weave" 
time="2016-04-20T15:39:20.534794228+02:00" level=error msg="Handler for GET /v1.18/images/weave/json returned error: No such image: weave" 
time="2016-04-20T15:39:21.199517004+02:00" level=error msg="Handler for GET /v1.18/containers/weavevolumes-1.5.0/json returned error: No such container: weavevolumes-1.5.0" 
time="2016-04-20T15:39:21.200007222+02:00" level=error msg="Handler for GET /v1.18/images/weavevolumes-1.5.0/json returned error: No such image: weavevolumes-1.5.0" 
time="2016-04-20T15:39:21.456626566+02:00" level=error msg="Handler for GET /v1.18/containers/weaveworks/weaveexec:1.5.0/json returned error: No such container: weaveworks/weaveexec:1.5.0" 
time="2016-04-20T15:39:21.475078185+02:00" level=error msg="Handler for GET /v1.18/containers/weaveproxy/json returned error: No such container: weaveproxy" 
time="2016-04-20T15:39:21.475494741+02:00" level=error msg="Handler for GET /v1.18/images/weaveproxy/json returned error: No such image: weaveproxy" 
time="2016-04-20T15:39:21.494102730+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T15:39:21.812039269+02:00" level=error msg="Handler for GET /v1.18/containers/weaveworks/plugin:1.5.0/json returned error: No such container: weaveworks/plugin:1.5.0" 
time="2016-04-20T15:39:21.831283696+02:00" level=error msg="Handler for GET /v1.18/containers/weaveplugin/json returned error: No such container: weaveplugin" 
time="2016-04-20T15:39:21.831730868+02:00" level=error msg="Handler for GET /v1.18/images/weaveplugin/json returned error: No such image: weaveplugin" 
time="2016-04-20T15:40:04.310392296+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T15:41:28.937650124+02:00" level=error msg="Handler for POST /v1.23/containers/a1/stop returned error: Container a1 is already stopped" 
time="2016-04-20T15:41:57.104547130+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T15:41:57.228357614+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T15:41:57.302352114+02:00" level=error msg="Handler for POST /v1.23/containers/da22f9e514de/stop returned error: Container da22f9e514de is already stopped" 
time="2016-04-20T15:41:57.357621580+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T15:41:57.448881488+02:00" level=error msg="Handler for POST /v1.23/containers/82e6196682ba/stop returned error: Container 82e6196682ba is already stopped" 
time="2016-04-20T15:42:42.677409797+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T15:42:42.953418194+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/NetworkDriver.DeleteNetwork, retrying in 1s" 
time="2016-04-20T15:42:43.953726432+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/NetworkDriver.DeleteNetwork, retrying in 2s" 
time="2016-04-20T15:42:45.954177947+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/NetworkDriver.DeleteNetwork, retrying in 4s" 
time="2016-04-20T15:42:49.954658716+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/NetworkDriver.DeleteNetwork, retrying in 8s" 
time="2016-04-20T15:42:57.955074236+02:00" level=warning msg="driver error deleting network weave : Post http://%2Frun%2Fdocker%2Fplugins%2Fweavemesh.sock/NetworkDriver.DeleteNetwork: dial unix /run/docker/plugins/weavemesh.sock: connect: no such file or directory" 
time="2016-04-20T15:42:57.955328360+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/IpamDriver.ReleaseAddress, retrying in 1s" 
time="2016-04-20T15:42:58.955755566+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/IpamDriver.ReleaseAddress, retrying in 2s" 
time="2016-04-20T15:43:00.956198956+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/IpamDriver.ReleaseAddress, retrying in 4s" 
time="2016-04-20T15:43:02.154948797+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T15:43:04.956708511+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/IpamDriver.ReleaseAddress, retrying in 8s" 
time="2016-04-20T15:43:05.061777609+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T15:43:05.322645847+02:00" level=error msg="Handler for DELETE /v1.21/networks/weave returned error: network weave not found" 
time="2016-04-20T15:43:05.348192217+02:00" level=error msg="Error setting up exec command in container weaveproxy: No such container: weaveproxy" 
time="2016-04-20T15:43:05.348240598+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/exec returned error: No such container: weaveproxy" 
time="2016-04-20T15:43:05.381953733+02:00" level=error msg="Handler for GET /v1.18/containers/weave/json returned error: No such container: weave" 
time="2016-04-20T15:43:05.382566382+02:00" level=error msg="Handler for GET /v1.18/images/weave/json returned error: No such image: weave" 
time="2016-04-20T15:43:05.405454062+02:00" level=error msg="Handler for POST /v1.18/containers/weaveplugin/stop returned error: No such container: weaveplugin" 
time="2016-04-20T15:43:05.423873367+02:00" level=error msg="Handler for DELETE /v1.18/containers/weaveplugin returned error: No such container: weaveplugin" 
time="2016-04-20T15:43:05.440708635+02:00" level=error msg="Handler for POST /v1.18/containers/weave/stop returned error: No such container: weave" 
time="2016-04-20T15:43:05.457553536+02:00" level=error msg="Handler for DELETE /v1.18/containers/weave returned error: No such container: weave" 
time="2016-04-20T15:43:05.474899874+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/stop returned error: No such container: weaveproxy" 
time="2016-04-20T15:43:05.492076755+02:00" level=error msg="Handler for DELETE /v1.18/containers/weaveproxy returned error: No such container: weaveproxy" 
time="2016-04-20T15:43:12.957287696+02:00" level=warning msg="Failed to release gateway ip address 10.32.0.0 on delete of network weave (59a0def179b50430fb59cf951a48cc1b583449836827f303e34d06d7cca28044): Post http://%2Frun%2Fdocker%2Fplugins%2Fweavemesh.sock/IpamDriver.ReleaseAddress: dial unix /run/docker/plugins/weavemesh.sock: connect: no such file or directory" 
time="2016-04-20T15:43:12.957526793+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/IpamDriver.ReleasePool, retrying in 1s" 
time="2016-04-20T15:43:13.957867633+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/IpamDriver.ReleasePool, retrying in 2s" 
time="2016-04-20T15:43:15.958314875+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/IpamDriver.ReleasePool, retrying in 4s" 
time="2016-04-20T15:43:19.958768392+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/IpamDriver.ReleasePool, retrying in 8s" 
time="2016-04-20T15:43:27.959208411+02:00" level=warning msg="Failed to release address pool weave-10.32.0.0/12-10.32.0.0/12 on delete of network weave (59a0def179b50430fb59cf951a48cc1b583449836827f303e34d06d7cca28044): Post http://%2Frun%2Fdocker%2Fplugins%2Fweavemesh.sock/IpamDriver.ReleasePool: dial unix /run/docker/plugins/weavemesh.sock: connect: no such file or directory" 
time="2016-04-20T15:43:51.162032481+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T15:43:51.452173373+02:00" level=error msg="Handler for GET /v1.18/containers/weavedb/json returned error: No such container: weavedb" 
time="2016-04-20T15:43:51.452708247+02:00" level=error msg="Handler for GET /v1.18/images/weavedb/json returned error: No such image: weavedb" 
time="2016-04-20T15:43:51.586224631+02:00" level=error msg="Handler for GET /v1.18/containers/weaveworks/weave:1.5.0/json returned error: No such container: weaveworks/weave:1.5.0" 
time="2016-04-20T15:43:51.607317751+02:00" level=error msg="Handler for GET /v1.18/containers/weave/json returned error: No such container: weave" 
time="2016-04-20T15:43:51.607751442+02:00" level=error msg="Handler for GET /v1.18/images/weave/json returned error: No such image: weave" 
time="2016-04-20T15:43:52.291072724+02:00" level=error msg="Handler for GET /v1.18/containers/weavevolumes-1.5.0/json returned error: No such container: weavevolumes-1.5.0" 
time="2016-04-20T15:43:52.291653694+02:00" level=error msg="Handler for GET /v1.18/images/weavevolumes-1.5.0/json returned error: No such image: weavevolumes-1.5.0" 
time="2016-04-20T15:43:52.532852741+02:00" level=error msg="Handler for GET /v1.18/containers/weaveworks/weaveexec:1.5.0/json returned error: No such container: weaveworks/weaveexec:1.5.0" 
time="2016-04-20T15:43:52.551601807+02:00" level=error msg="Handler for GET /v1.18/containers/weaveproxy/json returned error: No such container: weaveproxy" 
time="2016-04-20T15:43:52.552008369+02:00" level=error msg="Handler for GET /v1.18/images/weaveproxy/json returned error: No such image: weaveproxy" 
time="2016-04-20T15:43:52.571272285+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T15:43:52.873434694+02:00" level=error msg="Handler for GET /v1.18/containers/weaveworks/plugin:1.5.0/json returned error: No such container: weaveworks/plugin:1.5.0" 
time="2016-04-20T15:43:52.893578189+02:00" level=error msg="Handler for GET /v1.18/containers/weaveplugin/json returned error: No such container: weaveplugin" 
time="2016-04-20T15:43:52.894243329+02:00" level=error msg="Handler for GET /v1.18/images/weaveplugin/json returned error: No such image: weaveplugin" 
time="2016-04-20T15:44:40.139187445+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T15:45:30.674500337+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T15:45:30.976139976+02:00" level=warning msg="Failed to release gateway ip address 10.32.0.0 on delete of network weave (5d64abe415245f3a5d85238d36ea7efb14d3b60fe2b76f96e63803b917b12447): IpamDriver.ReleaseAddress: 400 Bad Request: Delete: no addresses for 10.32.0.0\n" 
time="2016-04-20T15:45:31.070566806+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T15:45:31.238256026+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T15:45:31.483566155+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T15:45:45.471266553+02:00" level=error msg="Handler for POST /v1.23/containers/368dfc9f01c7/stop returned error: Container 368dfc9f01c7 is already stopped" 
time="2016-04-20T15:45:45.472025148+02:00" level=error msg="Handler for POST /v1.23/containers/fdb1f278aa66/stop returned error: Container fdb1f278aa66 is already stopped" 
time="2016-04-20T15:45:45.472587994+02:00" level=error msg="Handler for POST /v1.23/containers/fe035307b967/stop returned error: Container fe035307b967 is already stopped" 
time="2016-04-20T15:45:45.473270628+02:00" level=error msg="Handler for POST /v1.23/containers/9690d89668ca/stop returned error: Container 9690d89668ca is already stopped" 
time="2016-04-20T15:45:45.473958474+02:00" level=error msg="Handler for POST /v1.23/containers/2b68e10ac0bb/stop returned error: Container 2b68e10ac0bb is already stopped" 
time="2016-04-20T15:45:45.474684163+02:00" level=error msg="Handler for POST /v1.23/containers/f948f99d7cf7/stop returned error: Container f948f99d7cf7 is already stopped" 
time="2016-04-20T15:46:19.689947409+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T15:46:19.940167031+02:00" level=error msg="Handler for DELETE /v1.21/networks/weave returned error: network weave not found" 
time="2016-04-20T15:46:19.957916884+02:00" level=error msg="Error setting up exec command in container weaveproxy: No such container: weaveproxy" 
time="2016-04-20T15:46:19.957949552+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/exec returned error: No such container: weaveproxy" 
time="2016-04-20T15:46:19.982195706+02:00" level=error msg="Handler for GET /v1.18/containers/weave/json returned error: No such container: weave" 
time="2016-04-20T15:46:19.982600301+02:00" level=error msg="Handler for GET /v1.18/images/weave/json returned error: No such image: weave" 
time="2016-04-20T15:46:19.999954361+02:00" level=error msg="Handler for POST /v1.18/containers/weaveplugin/stop returned error: No such container: weaveplugin" 
time="2016-04-20T15:46:20.016814762+02:00" level=error msg="Handler for DELETE /v1.18/containers/weaveplugin returned error: No such container: weaveplugin" 
time="2016-04-20T15:46:20.033534921+02:00" level=error msg="Handler for POST /v1.18/containers/weave/stop returned error: No such container: weave" 
time="2016-04-20T15:46:20.050303287+02:00" level=error msg="Handler for DELETE /v1.18/containers/weave returned error: No such container: weave" 
time="2016-04-20T15:46:20.067628009+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/stop returned error: No such container: weaveproxy" 
time="2016-04-20T15:46:20.084884593+02:00" level=error msg="Handler for DELETE /v1.18/containers/weaveproxy returned error: No such container: weaveproxy" 
time="2016-04-20T15:46:48.622310747+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T15:46:48.904421657+02:00" level=error msg="Handler for GET /v1.18/containers/weavedb/json returned error: No such container: weavedb" 
time="2016-04-20T15:46:48.904979139+02:00" level=error msg="Handler for GET /v1.18/images/weavedb/json returned error: No such image: weavedb" 
time="2016-04-20T15:46:49.022455096+02:00" level=error msg="Handler for GET /v1.18/containers/weaveworks/weave:1.5.0/json returned error: No such container: weaveworks/weave:1.5.0" 
time="2016-04-20T15:46:49.040651697+02:00" level=error msg="Handler for GET /v1.18/containers/weave/json returned error: No such container: weave" 
time="2016-04-20T15:46:49.041040435+02:00" level=error msg="Handler for GET /v1.18/images/weave/json returned error: No such image: weave" 
time="2016-04-20T15:46:49.706091649+02:00" level=error msg="Handler for GET /v1.18/containers/weavevolumes-1.5.0/json returned error: No such container: weavevolumes-1.5.0" 
time="2016-04-20T15:46:49.706523739+02:00" level=error msg="Handler for GET /v1.18/images/weavevolumes-1.5.0/json returned error: No such image: weavevolumes-1.5.0" 
time="2016-04-20T15:46:49.952857762+02:00" level=error msg="Handler for GET /v1.18/containers/weaveworks/weaveexec:1.5.0/json returned error: No such container: weaveworks/weaveexec:1.5.0" 
time="2016-04-20T15:46:49.971343597+02:00" level=error msg="Handler for GET /v1.18/containers/weaveproxy/json returned error: No such container: weaveproxy" 
time="2016-04-20T15:46:49.971809901+02:00" level=error msg="Handler for GET /v1.18/images/weaveproxy/json returned error: No such image: weaveproxy" 
time="2016-04-20T15:46:49.991526603+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T15:46:50.293468037+02:00" level=error msg="Handler for GET /v1.18/containers/weaveworks/plugin:1.5.0/json returned error: No such container: weaveworks/plugin:1.5.0" 
time="2016-04-20T15:46:50.312195381+02:00" level=error msg="Handler for GET /v1.18/containers/weaveplugin/json returned error: No such container: weaveplugin" 
time="2016-04-20T15:46:50.312580350+02:00" level=error msg="Handler for GET /v1.18/images/weaveplugin/json returned error: No such image: weaveplugin" 
time="2016-04-20T15:47:27.518952039+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T15:58:25.584367090+02:00" level=error msg="Handler for POST /v1.23/containers/66570043d312/stop returned error: Container 66570043d312 is already stopped" 
time="2016-04-20T15:58:25.628083053+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T15:58:25.736402825+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T15:58:25.807886989+02:00" level=error msg="Handler for POST /v1.23/containers/b0041a2be37f/stop returned error: Container b0041a2be37f is already stopped" 
time="2016-04-20T15:58:25.866199783+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T15:58:25.934070423+02:00" level=error msg="Handler for POST /v1.23/containers/9573fe45959b/stop returned error: Container 9573fe45959b is already stopped" 
time="2016-04-20T15:59:07.525412407+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T15:59:07.525412241+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T15:59:07.795893873+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/NetworkDriver.DeleteNetwork, retrying in 1s" 
time="2016-04-20T15:59:07.803737623+02:00" level=error msg="Handler for DELETE /v1.21/networks/weave returned error: network weave not found" 
time="2016-04-20T15:59:07.820513291+02:00" level=error msg="Handler for POST /v1.18/containers/weaveplugin/stop returned error: No such container: weaveplugin" 
time="2016-04-20T15:59:07.837488330+02:00" level=error msg="Handler for POST /v1.18/containers/weave/stop returned error: No such container: weave" 
time="2016-04-20T15:59:07.857982592+02:00" level=error msg="Error setting up exec command in container weaveproxy: No such container: weaveproxy" 
time="2016-04-20T15:59:07.858025131+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/exec returned error: No such container: weaveproxy" 
time="2016-04-20T15:59:07.875814815+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/stop returned error: No such container: weaveproxy" 
time="2016-04-20T15:59:08.796369279+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/NetworkDriver.DeleteNetwork, retrying in 2s" 
time="2016-04-20T15:59:10.741054544+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T15:59:10.796917226+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/NetworkDriver.DeleteNetwork, retrying in 4s" 
time="2016-04-20T15:59:13.541638426+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T15:59:13.541651175+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T15:59:13.821488098+02:00" level=error msg="Handler for DELETE /v1.21/networks/weave returned error: network weave not found" 
time="2016-04-20T15:59:13.834279347+02:00" level=error msg="Handler for DELETE /v1.21/networks/weave returned error: network weave not found" 
time="2016-04-20T15:59:13.840068529+02:00" level=error msg="Error setting up exec command in container weaveproxy: No such container: weaveproxy" 
time="2016-04-20T15:59:13.840113366+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/exec returned error: No such container: weaveproxy" 
time="2016-04-20T15:59:13.852131498+02:00" level=error msg="Error setting up exec command in container weaveproxy: No such container: weaveproxy" 
time="2016-04-20T15:59:13.852168210+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/exec returned error: No such container: weaveproxy" 
time="2016-04-20T15:59:13.863933349+02:00" level=error msg="Handler for GET /v1.18/containers/weave/json returned error: No such container: weave" 
time="2016-04-20T15:59:13.864392430+02:00" level=error msg="Handler for GET /v1.18/images/weave/json returned error: No such image: weave" 
time="2016-04-20T15:59:13.875969111+02:00" level=error msg="Handler for GET /v1.18/containers/weave/json returned error: No such container: weave" 
time="2016-04-20T15:59:13.876350847+02:00" level=error msg="Handler for GET /v1.18/images/weave/json returned error: No such image: weave" 
time="2016-04-20T15:59:13.882313695+02:00" level=error msg="Handler for POST /v1.18/containers/weaveplugin/stop returned error: No such container: weaveplugin" 
time="2016-04-20T15:59:13.893809329+02:00" level=error msg="Handler for POST /v1.18/containers/weaveplugin/stop returned error: No such container: weaveplugin" 
time="2016-04-20T15:59:13.899201684+02:00" level=error msg="Handler for DELETE /v1.18/containers/weaveplugin returned error: No such container: weaveplugin" 
time="2016-04-20T15:59:13.910708740+02:00" level=error msg="Handler for DELETE /v1.18/containers/weaveplugin returned error: No such container: weaveplugin" 
time="2016-04-20T15:59:13.915972739+02:00" level=error msg="Handler for POST /v1.18/containers/weave/stop returned error: No such container: weave" 
time="2016-04-20T15:59:13.927792061+02:00" level=error msg="Handler for POST /v1.18/containers/weave/stop returned error: No such container: weave" 
time="2016-04-20T15:59:13.932813720+02:00" level=error msg="Handler for DELETE /v1.18/containers/weave returned error: No such container: weave" 
time="2016-04-20T15:59:13.944647727+02:00" level=error msg="Handler for DELETE /v1.18/containers/weave returned error: No such container: weave" 
time="2016-04-20T15:59:13.949541933+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/stop returned error: No such container: weaveproxy" 
time="2016-04-20T15:59:13.961317113+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/stop returned error: No such container: weaveproxy" 
time="2016-04-20T15:59:13.966479054+02:00" level=error msg="Handler for DELETE /v1.18/containers/weaveproxy returned error: No such container: weaveproxy" 
time="2016-04-20T15:59:13.981628213+02:00" level=error msg="Handler for DELETE /v1.18/containers/weaveproxy returned error: No such container: weaveproxy" 
time="2016-04-20T15:59:14.797315126+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/NetworkDriver.DeleteNetwork, retrying in 8s" 
time="2016-04-20T15:59:22.797776894+02:00" level=warning msg="driver error deleting network weave : Post http://%2Frun%2Fdocker%2Fplugins%2Fweavemesh.sock/NetworkDriver.DeleteNetwork: dial unix /run/docker/plugins/weavemesh.sock: connect: no such file or directory" 
time="2016-04-20T15:59:22.798035512+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/IpamDriver.ReleaseAddress, retrying in 1s" 
time="2016-04-20T15:59:23.798391061+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/IpamDriver.ReleaseAddress, retrying in 2s" 
time="2016-04-20T15:59:25.798895113+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/IpamDriver.ReleaseAddress, retrying in 4s" 
time="2016-04-20T15:59:29.799298481+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/IpamDriver.ReleaseAddress, retrying in 8s" 
time="2016-04-20T15:59:37.799817180+02:00" level=warning msg="Failed to release gateway ip address 10.32.0.0 on delete of network weave (c7b022b458c3475eb6641b3e46a4a51ee93383e6d3a226147a23b0f4f041c21f): Post http://%2Frun%2Fdocker%2Fplugins%2Fweavemesh.sock/IpamDriver.ReleaseAddress: dial unix /run/docker/plugins/weavemesh.sock: connect: no such file or directory" 
time="2016-04-20T15:59:37.800077464+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/IpamDriver.ReleasePool, retrying in 1s" 
time="2016-04-20T15:59:38.800474315+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/IpamDriver.ReleasePool, retrying in 2s" 
time="2016-04-20T15:59:40.800885771+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/IpamDriver.ReleasePool, retrying in 4s" 
time="2016-04-20T15:59:44.801289315+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/IpamDriver.ReleasePool, retrying in 8s" 
time="2016-04-20T15:59:52.801663913+02:00" level=warning msg="Failed to release address pool weave-10.32.0.0/12-10.32.0.0/12 on delete of network weave (c7b022b458c3475eb6641b3e46a4a51ee93383e6d3a226147a23b0f4f041c21f): Post http://%2Frun%2Fdocker%2Fplugins%2Fweavemesh.sock/IpamDriver.ReleasePool: dial unix /run/docker/plugins/weavemesh.sock: connect: no such file or directory" 
time="2016-04-20T15:59:53.291504656+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T15:59:53.555339189+02:00" level=error msg="Handler for DELETE /v1.21/networks/weave returned error: network weave not found" 
time="2016-04-20T15:59:53.574049971+02:00" level=error msg="Error setting up exec command in container weaveproxy: No such container: weaveproxy" 
time="2016-04-20T15:59:53.574087873+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/exec returned error: No such container: weaveproxy" 
time="2016-04-20T15:59:53.597332927+02:00" level=error msg="Handler for GET /v1.18/containers/weave/json returned error: No such container: weave" 
time="2016-04-20T15:59:53.597742247+02:00" level=error msg="Handler for GET /v1.18/images/weave/json returned error: No such image: weave" 
time="2016-04-20T15:59:53.615292858+02:00" level=error msg="Handler for POST /v1.18/containers/weaveplugin/stop returned error: No such container: weaveplugin" 
time="2016-04-20T15:59:53.632105622+02:00" level=error msg="Handler for DELETE /v1.18/containers/weaveplugin returned error: No such container: weaveplugin" 
time="2016-04-20T15:59:53.648891733+02:00" level=error msg="Handler for POST /v1.18/containers/weave/stop returned error: No such container: weave" 
time="2016-04-20T15:59:53.665683485+02:00" level=error msg="Handler for DELETE /v1.18/containers/weave returned error: No such container: weave" 
time="2016-04-20T15:59:53.682650400+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/stop returned error: No such container: weaveproxy" 
time="2016-04-20T15:59:53.699420864+02:00" level=error msg="Handler for DELETE /v1.18/containers/weaveproxy returned error: No such container: weaveproxy" 
time="2016-04-20T16:00:21.664283213+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T16:00:21.967327987+02:00" level=error msg="Handler for GET /v1.18/containers/weavedb/json returned error: No such container: weavedb" 
time="2016-04-20T16:00:21.967887613+02:00" level=error msg="Handler for GET /v1.18/images/weavedb/json returned error: No such image: weavedb" 
time="2016-04-20T16:00:22.075700310+02:00" level=error msg="Handler for GET /v1.18/containers/weaveworks/weave:1.5.0/json returned error: No such container: weaveworks/weave:1.5.0" 
time="2016-04-20T16:00:22.094502115+02:00" level=error msg="Handler for GET /v1.18/containers/weave/json returned error: No such container: weave" 
time="2016-04-20T16:00:22.094907302+02:00" level=error msg="Handler for GET /v1.18/images/weave/json returned error: No such image: weave" 
time="2016-04-20T16:00:22.747999995+02:00" level=error msg="Handler for GET /v1.18/containers/weavevolumes-1.5.0/json returned error: No such container: weavevolumes-1.5.0" 
time="2016-04-20T16:00:22.748504474+02:00" level=error msg="Handler for GET /v1.18/images/weavevolumes-1.5.0/json returned error: No such image: weavevolumes-1.5.0" 
time="2016-04-20T16:00:22.996239623+02:00" level=error msg="Handler for GET /v1.18/containers/weaveworks/weaveexec:1.5.0/json returned error: No such container: weaveworks/weaveexec:1.5.0" 
time="2016-04-20T16:00:23.015530092+02:00" level=error msg="Handler for GET /v1.18/containers/weaveproxy/json returned error: No such container: weaveproxy" 
time="2016-04-20T16:00:23.015926524+02:00" level=error msg="Handler for GET /v1.18/images/weaveproxy/json returned error: No such image: weaveproxy" 
time="2016-04-20T16:00:23.035576634+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T16:00:23.373484407+02:00" level=error msg="Handler for GET /v1.18/containers/weaveworks/plugin:1.5.0/json returned error: No such container: weaveworks/plugin:1.5.0" 
time="2016-04-20T16:00:23.393133788+02:00" level=error msg="Handler for GET /v1.18/containers/weaveplugin/json returned error: No such container: weaveplugin" 
time="2016-04-20T16:00:23.393492812+02:00" level=error msg="Handler for GET /v1.18/images/weaveplugin/json returned error: No such image: weaveplugin" 
time="2016-04-20T16:00:32.810784607+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T16:00:56.172893487+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T16:01:10.901019600+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T16:01:23.710245610+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T16:12:11.500921889+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T16:42:37.441152540+02:00" level=error msg="Handler for POST /v1.23/containers/949b520b8bea/stop returned error: Container 949b520b8bea is already stopped" 
time="2016-04-20T16:42:37.441706628+02:00" level=error msg="Handler for POST /v1.23/containers/949b520b8bea/stop returned error: Container 949b520b8bea is already stopped" 
time="2016-04-20T16:42:37.484837492+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T16:42:37.610965353+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T16:42:37.687323589+02:00" level=error msg="Handler for POST /v1.23/containers/9799458602f5/stop returned error: Container 9799458602f5 is already stopped" 
time="2016-04-20T16:42:37.687342476+02:00" level=error msg="Handler for POST /v1.23/containers/9799458602f5/stop returned error: Container 9799458602f5 is already stopped" 
time="2016-04-20T16:42:37.747845852+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T16:42:37.814035608+02:00" level=error msg="Handler for POST /v1.23/containers/927762e9aca3/stop returned error: Container 927762e9aca3 is already stopped" 
time="2016-04-20T16:42:37.814035693+02:00" level=error msg="Handler for POST /v1.23/containers/927762e9aca3/stop returned error: Container 927762e9aca3 is already stopped" 
time="2016-04-20T16:42:37.902037321+02:00" level=error msg="Handler for DELETE /v1.23/containers/949b520b8bea returned error: No such container: 949b520b8bea" 
time="2016-04-20T16:42:37.909736320+02:00" level=error msg="Handler for DELETE /v1.23/containers/9c7e6fd5b1f9 returned error: No such container: 9c7e6fd5b1f9" 
time="2016-04-20T16:42:37.910682631+02:00" level=error msg="Handler for DELETE /v1.23/containers/docker returned error: No such container: docker" 
time="2016-04-20T16:42:37.911122550+02:00" level=error msg="Handler for DELETE /v1.23/containers/rmi returned error: No such container: rmi" 
time="2016-04-20T16:42:37.911688058+02:00" level=error msg="Handler for DELETE /v1.23/containers/docker returned error: No such container: docker" 
time="2016-04-20T16:42:37.912218915+02:00" level=error msg="Handler for DELETE /v1.23/containers/rmi returned error: No such container: rmi" 
time="2016-04-20T16:44:00.583944541+02:00" level=error msg="Handler for DELETE /v1.23/images/weaveworks/weave returned error: No such image: weaveworks/weave:latest" 
time="2016-04-20T16:44:00.585196571+02:00" level=error msg="Handler for DELETE /v1.23/images/weaveworks/weave returned error: No such image: weaveworks/weave:latest" 
time="2016-04-20T16:44:24.175592784+02:00" level=error msg="Handler for DELETE /v1.23/images/weaveworks/weave:1.5.0 returned error: No such image: weaveworks/weave:1.5.0" 
time="2016-04-20T16:44:24.362114879+02:00" level=error msg="Handler for DELETE /v1.23/images/weaveworks/weavedb:1.5.0 returned error: No such image: weaveworks/weavedb:1.5.0" 
time="2016-04-20T16:44:47.100129463+02:00" level=error msg="Handler for DELETE /v1.23/images/weaveworks/weavedb returned error: reference does not exist" 
time="2016-04-20T16:45:29.619260862+02:00" level=error msg="Handler for POST /v1.23/containers/create returned error: No such image: weaveworks/weaveexec:1.5.0" 
time="2016-04-20T16:45:29.619321749+02:00" level=error msg="Handler for POST /v1.23/containers/create returned error: No such image: weaveworks/weaveexec:1.5.0" 
time="2016-04-20T16:45:34.169606112+02:00" level=info msg="Pull session cancelled" 
time="2016-04-20T16:45:34.169599290+02:00" level=info msg="Pull session cancelled" 
time="2016-04-20T16:45:34.169726712+02:00" level=error msg="Error trying v2 registry: context canceled" 
time="2016-04-20T16:45:34.169769717+02:00" level=error msg="Not continuing with pull after error: context canceled" 
time="2016-04-20T16:45:34.169796718+02:00" level=error msg="Error trying v2 registry: context canceled" 
time="2016-04-20T16:45:34.169838894+02:00" level=error msg="Not continuing with pull after error: context canceled" 
time="2016-04-20T16:45:41.962052368+02:00" level=error msg="Handler for DELETE /v1.23/images/weaveworks/weaveexec:1.5.0 returned error: No such image: weaveworks/weaveexec:1.5.0" 
time="2016-04-20T16:45:41.962225270+02:00" level=error msg="Handler for DELETE /v1.23/images/weaveworks/weaveexec:1.5.0 returned error: No such image: weaveworks/weaveexec:1.5.0" 
time="2016-04-20T16:47:26.398817368+02:00" level=error msg="Handler for POST /v1.23/containers/create returned error: No such image: weaveworks/weaveexec:1.5.0" 
time="2016-04-20T16:47:26.398967764+02:00" level=error msg="Handler for POST /v1.23/containers/create returned error: No such image: weaveworks/weaveexec:1.5.0" 
time="2016-04-20T16:47:28.496165253+02:00" level=info msg="Pull session cancelled" 
time="2016-04-20T16:47:28.496365606+02:00" level=error msg="Error trying v2 registry: context canceled" 
time="2016-04-20T16:47:28.496391442+02:00" level=error msg="Not continuing with pull after error: context canceled" 
time="2016-04-20T16:47:36.821383015+02:00" level=warning msg="Error getting v2 registry: Get https://registry-1.docker.io/v2/: net/http: TLS handshake timeout" 
time="2016-04-20T16:47:36.821469893+02:00" level=error msg="Attempting next endpoint for pull after error: Get https://registry-1.docker.io/v2/: net/http: TLS handshake timeout" 
time="2016-04-20T16:47:38.430633090+02:00" level=info msg="Pull session cancelled" 
time="2016-04-20T16:47:45.221731260+02:00" level=error msg="Not continuing with pull after error: Error pulling image (1.5.0) from docker.io/weaveworks/weaveexec, context canceled" 
time="2016-04-20T16:51:17.590418245+02:00" level=error msg="Handler for POST /v1.23/containers/create returned error: No such image: weaveworks/weaveexec:1.4.6" 
time="2016-04-20T16:51:29.913905850+02:00" level=error msg="Handler for POST /v1.18/containers/create returned error: No such image: weaveworks/plugin:1.4.6" 
time="2016-04-20T16:51:33.067964435+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/NetworkDriver.DeleteNetwork, retrying in 1s" 
time="2016-04-20T16:51:34.068335726+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/NetworkDriver.DeleteNetwork, retrying in 2s" 
time="2016-04-20T16:51:36.068748278+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/NetworkDriver.DeleteNetwork, retrying in 4s" 
time="2016-04-20T16:51:40.069127725+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/NetworkDriver.DeleteNetwork, retrying in 8s" 
time="2016-04-20T16:51:48.069567562+02:00" level=warning msg="driver error deleting network weave : Post http://%2Frun%2Fdocker%2Fplugins%2Fweavemesh.sock/NetworkDriver.DeleteNetwork: dial unix /run/docker/plugins/weavemesh.sock: connect: no such file or directory" 
time="2016-04-20T16:51:48.069775045+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/IpamDriver.ReleaseAddress, retrying in 1s" 
time="2016-04-20T16:51:49.070161737+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/IpamDriver.ReleaseAddress, retrying in 2s" 
time="2016-04-20T16:51:51.070523520+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/IpamDriver.ReleaseAddress, retrying in 4s" 
time="2016-04-20T16:51:55.070954183+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/IpamDriver.ReleaseAddress, retrying in 8s" 
time="2016-04-20T16:52:03.071416658+02:00" level=warning msg="Failed to release gateway ip address 10.32.0.0 on delete of network weave (794872169fe402bf13cb732517d2d64fff062f3b454fdd708d3c8a8e404e9318): Post http://%2Frun%2Fdocker%2Fplugins%2Fweavemesh.sock/IpamDriver.ReleaseAddress: dial unix /run/docker/plugins/weavemesh.sock: connect: no such file or directory" 
time="2016-04-20T16:52:03.071731010+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/IpamDriver.ReleasePool, retrying in 1s" 
time="2016-04-20T16:52:04.072199627+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/IpamDriver.ReleasePool, retrying in 2s" 
time="2016-04-20T16:52:06.072555192+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/IpamDriver.ReleasePool, retrying in 4s" 
time="2016-04-20T16:52:10.072965779+02:00" level=warning msg="Unable to connect to plugin: /run/docker/plugins/weavemesh.sock:/IpamDriver.ReleasePool, retrying in 8s" 
time="2016-04-20T16:52:18.073357441+02:00" level=warning msg="Failed to release address pool weave-10.32.0.0/12-10.32.0.0/12 on delete of network weave (794872169fe402bf13cb732517d2d64fff062f3b454fdd708d3c8a8e404e9318): Post http://%2Frun%2Fdocker%2Fplugins%2Fweavemesh.sock/IpamDriver.ReleasePool: dial unix /run/docker/plugins/weavemesh.sock: connect: no such file or directory" 
time="2016-04-20T16:52:18.243549421+02:00" level=error msg="Error setting up exec command in container weaveproxy: No such container: weaveproxy" 
time="2016-04-20T16:52:18.243624128+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/exec returned error: No such container: weaveproxy" 
time="2016-04-20T16:52:18.267866231+02:00" level=error msg="Handler for GET /v1.18/containers/weave/json returned error: No such container: weave" 
time="2016-04-20T16:52:18.268430563+02:00" level=error msg="Handler for GET /v1.18/images/weave/json returned error: No such image: weave" 
time="2016-04-20T16:52:18.289876111+02:00" level=error msg="Handler for POST /v1.18/containers/weaveplugin/stop returned error: No such container: weaveplugin" 
time="2016-04-20T16:52:18.308603314+02:00" level=error msg="Handler for DELETE /v1.18/containers/weaveplugin returned error: No such container: weaveplugin" 
time="2016-04-20T16:52:18.326034954+02:00" level=error msg="Handler for POST /v1.18/containers/weave/stop returned error: No such container: weave" 
time="2016-04-20T16:52:18.343049996+02:00" level=error msg="Handler for DELETE /v1.18/containers/weave returned error: No such container: weave" 
time="2016-04-20T16:52:18.360050224+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/stop returned error: No such container: weaveproxy" 
time="2016-04-20T16:52:18.377055093+02:00" level=error msg="Handler for DELETE /v1.18/containers/weaveproxy returned error: No such container: weaveproxy" 
time="2016-04-20T16:52:18.414971539+02:00" level=error msg="Handler for POST /v1.18/containers/create returned error: No such image: weaveworks/weave:1.4.6" 
time="2016-04-20T16:52:29.596728737+02:00" level=error msg="Handler for GET /v1.21/networks/weave returned error: network weave not found" 
time="2016-04-20T16:52:29.742181905+02:00" level=error msg="Error setting up exec command in container weaveproxy: No such container: weaveproxy" 
time="2016-04-20T16:52:29.742261097+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/exec returned error: No such container: weaveproxy" 
time="2016-04-20T16:52:29.766391703+02:00" level=error msg="Handler for GET /v1.18/containers/weave/json returned error: No such container: weave" 
time="2016-04-20T16:52:29.766830173+02:00" level=error msg="Handler for GET /v1.18/images/weave/json returned error: No such image: weave" 
time="2016-04-20T16:52:29.785654034+02:00" level=error msg="Handler for POST /v1.18/containers/weaveplugin/stop returned error: No such container: weaveplugin" 
time="2016-04-20T16:52:29.803004375+02:00" level=error msg="Handler for DELETE /v1.18/containers/weaveplugin returned error: No such container: weaveplugin" 
time="2016-04-20T16:52:29.820038429+02:00" level=error msg="Handler for POST /v1.18/containers/weave/stop returned error: No such container: weave" 
time="2016-04-20T16:52:29.837006921+02:00" level=error msg="Handler for DELETE /v1.18/containers/weave returned error: No such container: weave" 
time="2016-04-20T16:52:29.854017596+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/stop returned error: No such container: weaveproxy" 
time="2016-04-20T16:52:29.870891470+02:00" level=error msg="Handler for DELETE /v1.18/containers/weaveproxy returned error: No such container: weaveproxy" 
time="2016-04-20T16:52:57.149144133+02:00" level=error msg="Handler for GET /v1.18/containers/weave/json returned error: No such container: weave" 
time="2016-04-20T16:52:57.149740238+02:00" level=error msg="Handler for GET /v1.18/images/weave/json returned error: No such image: weave" 
time="2016-04-20T16:54:13.710236504+02:00" level=error msg="Handler for GET /v1.18/containers/weave/json returned error: No such container: weave" 
time="2016-04-20T16:54:13.710732883+02:00" level=error msg="Handler for GET /v1.18/images/weave/json returned error: No such image: weave" 
time="2016-04-20T16:54:13.727436359+02:00" level=error msg="Handler for GET /v1.18/containers/weaveproxy/json returned error: No such container: weaveproxy" 
time="2016-04-20T16:54:13.727782571+02:00" level=error msg="Handler for GET /v1.18/images/weaveproxy/json returned error: No such image: weaveproxy" 
time="2016-04-20T16:54:15.763027158+02:00" level=error msg="Handler for GET /v1.18/containers/weavevolumes-1.4.6/json returned error: No such container: weavevolumes-1.4.6" 
time="2016-04-20T16:54:15.763556103+02:00" level=error msg="Handler for GET /v1.18/images/weavevolumes-1.4.6/json returned error: No such image: weavevolumes-1.4.6" 
time="2016-04-20T16:54:16.339067103+02:00" level=error msg="Handler for GET /v1.18/containers/weaveplugin/json returned error: No such container: weaveplugin" 
time="2016-04-20T16:54:16.339511549+02:00" level=error msg="Handler for GET /v1.18/images/weaveplugin/json returned error: No such image: weaveplugin" 
time="2016-04-20T16:54:16.516220003+02:00" level=error msg="Handler for GET /v1.21/networks/weave returned error: network weave not found" 
time="2016-04-20T16:55:54.160288913+02:00" level=warning msg="Failed to release ip address 10.40.0.0 on delete of endpoint a1 (d7809c5000dc3bd6dfd6fa588f5cf464bb15e2f124abbbc09e8ea91849b2d58d): IpamDriver.ReleaseAddress: invalid CIDR address: " 
time="2016-04-20T16:56:33.954987337+02:00" level=warning msg="Failed to release gateway ip address 10.32.0.0 on delete of network weave (e68844c2b796c0e7dae90ffe0fc68ab2d239c76c7fdce565fcc07b5bae5d4a74): IpamDriver.ReleaseAddress: 400 Bad Request: Delete: no addresses for 10.32.0.0\n" 
time="2016-04-20T16:56:34.158596147+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T16:56:34.338648104+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T16:56:34.601521548+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T16:56:45.345806733+02:00" level=error msg="Handler for GET /v1.21/networks/weave returned error: network weave not found" 
time="2016-04-20T16:56:45.490359354+02:00" level=error msg="Error setting up exec command in container weaveproxy: No such container: weaveproxy" 
time="2016-04-20T16:56:45.490438366+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/exec returned error: No such container: weaveproxy" 
time="2016-04-20T16:56:45.511064258+02:00" level=error msg="Handler for GET /v1.18/containers/weave/json returned error: No such container: weave" 
time="2016-04-20T16:56:45.511601288+02:00" level=error msg="Handler for GET /v1.18/images/weave/json returned error: No such image: weave" 
time="2016-04-20T16:56:45.530684403+02:00" level=error msg="Handler for POST /v1.18/containers/weaveplugin/stop returned error: No such container: weaveplugin" 
time="2016-04-20T16:56:45.553896512+02:00" level=error msg="Handler for DELETE /v1.18/containers/weaveplugin returned error: No such container: weaveplugin" 
time="2016-04-20T16:56:45.572961410+02:00" level=error msg="Handler for POST /v1.18/containers/weave/stop returned error: No such container: weave" 
time="2016-04-20T16:56:45.593266358+02:00" level=error msg="Handler for DELETE /v1.18/containers/weave returned error: No such container: weave" 
time="2016-04-20T16:56:45.611994959+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/stop returned error: No such container: weaveproxy" 
time="2016-04-20T16:56:45.628999178+02:00" level=error msg="Handler for DELETE /v1.18/containers/weaveproxy returned error: No such container: weaveproxy" 
time="2016-04-20T16:57:08.063210276+02:00" level=error msg="Handler for POST /v1.23/containers/a1/stop returned error: Container a1 is already stopped" 
time="2016-04-20T16:57:27.558031548+02:00" level=error msg="Handler for DELETE /v1.23/images/weaveworks/weave:1.5.0 returned error: No such image: weaveworks/weave:1.5.0" 
time="2016-04-20T16:59:14.882205873+02:00" level=error msg="Handler for POST /v1.23/containers/create returned error: No such image: weaveworks/weaveexec:1.5.0" 
time="2016-04-20T16:59:24.608856806+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T16:59:24.884426753+02:00" level=error msg="Handler for DELETE /v1.21/networks/weave returned error: network weave not found" 
time="2016-04-20T16:59:24.905079142+02:00" level=error msg="Error setting up exec command in container weaveproxy: No such container: weaveproxy" 
time="2016-04-20T16:59:24.905114861+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/exec returned error: No such container: weaveproxy" 
time="2016-04-20T16:59:24.928865249+02:00" level=error msg="Handler for GET /v1.18/containers/weave/json returned error: No such container: weave" 
time="2016-04-20T16:59:24.929321515+02:00" level=error msg="Handler for GET /v1.18/images/weave/json returned error: No such image: weave" 
time="2016-04-20T16:59:24.946860410+02:00" level=error msg="Handler for POST /v1.18/containers/weaveplugin/stop returned error: No such container: weaveplugin" 
time="2016-04-20T16:59:24.964075836+02:00" level=error msg="Handler for DELETE /v1.18/containers/weaveplugin returned error: No such container: weaveplugin" 
time="2016-04-20T16:59:24.981133958+02:00" level=error msg="Handler for POST /v1.18/containers/weave/stop returned error: No such container: weave" 
time="2016-04-20T16:59:24.997876108+02:00" level=error msg="Handler for DELETE /v1.18/containers/weave returned error: No such container: weave" 
time="2016-04-20T16:59:25.014632263+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/stop returned error: No such container: weaveproxy" 
time="2016-04-20T16:59:25.031689839+02:00" level=error msg="Handler for DELETE /v1.18/containers/weaveproxy returned error: No such container: weaveproxy" 
time="2016-04-20T17:00:06.037248601+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T17:00:06.285939308+02:00" level=error msg="Handler for DELETE /v1.21/networks/weave returned error: network weave not found" 
time="2016-04-20T17:00:06.303812998+02:00" level=error msg="Error setting up exec command in container weaveproxy: No such container: weaveproxy" 
time="2016-04-20T17:00:06.303851066+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/exec returned error: No such container: weaveproxy" 
time="2016-04-20T17:00:06.327799077+02:00" level=error msg="Handler for GET /v1.18/containers/weave/json returned error: No such container: weave" 
time="2016-04-20T17:00:06.328241289+02:00" level=error msg="Handler for GET /v1.18/images/weave/json returned error: No such image: weave" 
time="2016-04-20T17:00:06.345765187+02:00" level=error msg="Handler for POST /v1.18/containers/weaveplugin/stop returned error: No such container: weaveplugin" 
time="2016-04-20T17:00:06.362656002+02:00" level=error msg="Handler for DELETE /v1.18/containers/weaveplugin returned error: No such container: weaveplugin" 
time="2016-04-20T17:00:06.379865025+02:00" level=error msg="Handler for POST /v1.18/containers/weave/stop returned error: No such container: weave" 
time="2016-04-20T17:00:06.396684747+02:00" level=error msg="Handler for DELETE /v1.18/containers/weave returned error: No such container: weave" 
time="2016-04-20T17:00:06.413616218+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/stop returned error: No such container: weaveproxy" 
time="2016-04-20T17:00:06.430449352+02:00" level=error msg="Handler for DELETE /v1.18/containers/weaveproxy returned error: No such container: weaveproxy" 
time="2016-04-20T17:00:47.073591502+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T17:00:47.370366936+02:00" level=error msg="Handler for GET /v1.18/containers/weavedb/json returned error: No such container: weavedb" 
time="2016-04-20T17:00:47.370920632+02:00" level=error msg="Handler for GET /v1.18/images/weavedb/json returned error: No such image: weavedb" 
time="2016-04-20T17:00:47.389418830+02:00" level=error msg="Handler for POST /v1.18/containers/create returned error: No such image: weaveworks/weavedb:latest" 
time="2016-04-20T17:00:50.423135507+02:00" level=error msg="Handler for GET /v1.18/containers/weaveworks/weave:1.5.0/json returned error: No such container: weaveworks/weave:1.5.0" 
time="2016-04-20T17:00:50.423874735+02:00" level=error msg="Handler for GET /v1.18/images/weaveworks/weave:1.5.0/json returned error: No such image: weaveworks/weave:1.5.0" 
time="2016-04-20T17:00:50.444886142+02:00" level=error msg="Handler for GET /v1.18/containers/weave/json returned error: No such container: weave" 
time="2016-04-20T17:00:50.445342186+02:00" level=error msg="Handler for GET /v1.18/images/weave/json returned error: No such image: weave" 
time="2016-04-20T17:00:50.464266364+02:00" level=error msg="Handler for POST /v1.18/containers/create returned error: No such image: weaveworks/weave:1.5.0" 
time="2016-04-20T17:00:54.673167739+02:00" level=error msg="Handler for GET /v1.18/containers/weavevolumes-1.5.0/json returned error: No such container: weavevolumes-1.5.0" 
time="2016-04-20T17:00:54.673568971+02:00" level=error msg="Handler for GET /v1.18/images/weavevolumes-1.5.0/json returned error: No such image: weavevolumes-1.5.0" 
time="2016-04-20T17:00:54.908576453+02:00" level=error msg="Handler for GET /v1.18/containers/weaveworks/weaveexec:1.5.0/json returned error: No such container: weaveworks/weaveexec:1.5.0" 
time="2016-04-20T17:00:54.930383344+02:00" level=error msg="Handler for GET /v1.18/containers/weaveproxy/json returned error: No such container: weaveproxy" 
time="2016-04-20T17:00:54.930890846+02:00" level=error msg="Handler for GET /v1.18/images/weaveproxy/json returned error: No such image: weaveproxy" 
time="2016-04-20T17:00:54.950318541+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T17:00:55.274730768+02:00" level=error msg="Handler for GET /v1.18/containers/weaveworks/plugin:1.5.0/json returned error: No such container: weaveworks/plugin:1.5.0" 
time="2016-04-20T17:00:55.275135266+02:00" level=error msg="Handler for GET /v1.18/images/weaveworks/plugin:1.5.0/json returned error: No such image: weaveworks/plugin:1.5.0" 
time="2016-04-20T17:00:55.292618295+02:00" level=error msg="Handler for GET /v1.18/containers/weaveplugin/json returned error: No such container: weaveplugin" 
time="2016-04-20T17:00:55.293009216+02:00" level=error msg="Handler for GET /v1.18/images/weaveplugin/json returned error: No such image: weaveplugin" 
time="2016-04-20T17:00:55.311846588+02:00" level=error msg="Handler for POST /v1.18/containers/create returned error: No such image: weaveworks/plugin:1.5.0" 
time="2016-04-20T17:01:01.889536875+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T17:01:02.150222198+02:00" level=warning msg="Failed to release gateway ip address 10.32.0.0 on delete of network weave (bd1376ff860d4e9c559a775e97e416a2eb37592e1b58104c3f3314a82aa8fc89): IpamDriver.ReleaseAddress: 400 Bad Request: Delete: no addresses for 10.32.0.0\n" 
time="2016-04-20T17:01:02.337567948+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T17:01:02.536302572+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T17:01:02.693038065+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T17:01:09.217896414+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T17:01:09.517643966+02:00" level=error msg="Handler for GET /v1.18/containers/weavedb/json returned error: No such container: weavedb" 
time="2016-04-20T17:01:09.518216481+02:00" level=error msg="Handler for GET /v1.18/images/weavedb/json returned error: No such image: weavedb" 
time="2016-04-20T17:01:09.630759857+02:00" level=error msg="Handler for GET /v1.18/containers/weaveworks/weave:1.5.0/json returned error: No such container: weaveworks/weave:1.5.0" 
time="2016-04-20T17:01:09.649092164+02:00" level=error msg="Handler for GET /v1.18/containers/weave/json returned error: No such container: weave" 
time="2016-04-20T17:01:09.649457965+02:00" level=error msg="Handler for GET /v1.18/images/weave/json returned error: No such image: weave" 
time="2016-04-20T17:01:10.287840791+02:00" level=error msg="Handler for GET /v1.18/containers/weavevolumes-1.5.0/json returned error: No such container: weavevolumes-1.5.0" 
time="2016-04-20T17:01:10.288295327+02:00" level=error msg="Handler for GET /v1.18/images/weavevolumes-1.5.0/json returned error: No such image: weavevolumes-1.5.0" 
time="2016-04-20T17:01:10.538350251+02:00" level=error msg="Handler for GET /v1.18/containers/weaveworks/weaveexec:1.5.0/json returned error: No such container: weaveworks/weaveexec:1.5.0" 
time="2016-04-20T17:01:10.556950807+02:00" level=error msg="Handler for GET /v1.18/containers/weaveproxy/json returned error: No such container: weaveproxy" 
time="2016-04-20T17:01:10.557338506+02:00" level=error msg="Handler for GET /v1.18/images/weaveproxy/json returned error: No such image: weaveproxy" 
time="2016-04-20T17:01:10.575860263+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T17:01:10.880221313+02:00" level=error msg="Handler for GET /v1.18/containers/weaveworks/plugin:1.5.0/json returned error: No such container: weaveworks/plugin:1.5.0" 
time="2016-04-20T17:01:10.900469291+02:00" level=error msg="Handler for GET /v1.18/containers/weaveplugin/json returned error: No such container: weaveplugin" 
time="2016-04-20T17:01:10.900876304+02:00" level=error msg="Handler for GET /v1.18/images/weaveplugin/json returned error: No such image: weaveplugin" 
time="2016-04-20T17:01:43.912078213+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T17:02:13.229907163+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T17:02:28.593843635+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T17:02:53.155560183+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T17:02:53.435949667+02:00" level=warning msg="Failed to release gateway ip address 10.32.0.0 on delete of network weave (bbb74d9d2d525613a151bc3ebe00aeec25c9a2e04e072f13bf83dc4f41741a90): IpamDriver.ReleaseAddress: 400 Bad Request: Delete: no addresses for 10.32.0.0\n" 
time="2016-04-20T17:02:53.529374604+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T17:02:53.687953394+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T17:02:53.949358152+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T17:03:07.708097056+02:00" level=error msg="Handler for POST /v1.23/containers/a1/stop returned error: Container a1 is already stopped" 
time="2016-04-20T17:03:23.166860548+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T17:03:23.439892907+02:00" level=error msg="Handler for DELETE /v1.21/networks/weave returned error: network weave not found" 
time="2016-04-20T17:03:23.456918140+02:00" level=error msg="Handler for POST /v1.18/containers/weaveplugin/stop returned error: Container weaveplugin is already stopped" 
time="2016-04-20T17:03:23.473694938+02:00" level=error msg="Handler for POST /v1.18/containers/weave/stop returned error: Container weave is already stopped" 
time="2016-04-20T17:03:23.493114359+02:00" level=error msg="Error setting up exec command in container weaveproxy: Container 0019a1d744cf9b2cdac8712d939bc928982960c7ea94610311a435e9e1ec740f is not running" 
time="2016-04-20T17:03:23.493156220+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/exec returned error: Container 0019a1d744cf9b2cdac8712d939bc928982960c7ea94610311a435e9e1ec740f is not running" 
time="2016-04-20T17:03:23.510509610+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/stop returned error: Container weaveproxy is already stopped" 
time="2016-04-20T17:03:36.270371147+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T17:03:36.540813846+02:00" level=error msg="Handler for DELETE /v1.21/networks/weave returned error: network weave not found" 
time="2016-04-20T17:03:36.563683250+02:00" level=error msg="Error setting up exec command in container weaveproxy: Container 0019a1d744cf9b2cdac8712d939bc928982960c7ea94610311a435e9e1ec740f is not running" 
time="2016-04-20T17:03:36.563723141+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/exec returned error: Container 0019a1d744cf9b2cdac8712d939bc928982960c7ea94610311a435e9e1ec740f is not running" 
time="2016-04-20T17:03:36.606640790+02:00" level=error msg="Handler for POST /v1.18/containers/weaveplugin/stop returned error: Container weaveplugin is already stopped" 
time="2016-04-20T17:03:36.645150857+02:00" level=error msg="Handler for POST /v1.18/containers/weave/stop returned error: Container weave is already stopped" 
time="2016-04-20T17:03:36.680331848+02:00" level=error msg="Handler for POST /v1.18/containers/weaveproxy/stop returned error: Container weaveproxy is already stopped" 
time="2016-04-20T17:03:54.846014017+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T17:03:55.112197930+02:00" level=error msg="Handler for GET /v1.18/containers/weavedb/json returned error: No such container: weavedb" 
time="2016-04-20T17:03:55.112777954+02:00" level=error msg="Handler for GET /v1.18/images/weavedb/json returned error: No such image: weavedb" 
time="2016-04-20T17:03:55.233403621+02:00" level=error msg="Handler for GET /v1.18/containers/weaveworks/weave:1.5.0/json returned error: No such container: weaveworks/weave:1.5.0" 
time="2016-04-20T17:03:55.255238283+02:00" level=error msg="Handler for GET /v1.18/containers/weave/json returned error: No such container: weave" 
time="2016-04-20T17:03:55.255670619+02:00" level=error msg="Handler for GET /v1.18/images/weave/json returned error: No such image: weave" 
time="2016-04-20T17:03:55.908985750+02:00" level=error msg="Handler for GET /v1.18/containers/weavevolumes-1.5.0/json returned error: No such container: weavevolumes-1.5.0" 
time="2016-04-20T17:03:55.909481592+02:00" level=error msg="Handler for GET /v1.18/images/weavevolumes-1.5.0/json returned error: No such image: weavevolumes-1.5.0" 
time="2016-04-20T17:03:56.151846801+02:00" level=error msg="Handler for GET /v1.18/containers/weaveworks/weaveexec:1.5.0/json returned error: No such container: weaveworks/weaveexec:1.5.0" 
time="2016-04-20T17:03:56.173516953+02:00" level=error msg="Handler for GET /v1.18/containers/weaveproxy/json returned error: No such container: weaveproxy" 
time="2016-04-20T17:03:56.173964377+02:00" level=error msg="Handler for GET /v1.18/images/weaveproxy/json returned error: No such image: weaveproxy" 
time="2016-04-20T17:03:56.194501145+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T17:03:56.493157972+02:00" level=error msg="Handler for GET /v1.18/containers/weaveworks/plugin:1.5.0/json returned error: No such container: weaveworks/plugin:1.5.0" 
time="2016-04-20T17:03:56.514655136+02:00" level=error msg="Handler for GET /v1.18/containers/weaveplugin/json returned error: No such container: weaveplugin" 
time="2016-04-20T17:03:56.515083708+02:00" level=error msg="Handler for GET /v1.18/images/weaveplugin/json returned error: No such image: weaveplugin" 
time="2016-04-20T17:04:02.214569400+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T17:04:27.574056146+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T17:04:45.065067723+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T17:10:47.783737167+02:00" level=info msg="Processing signal 'terminated'" 
time="2016-04-20T17:10:47.841903453+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T17:10:47.867918354+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T17:10:47.891795032+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T17:10:48.004288833+02:00" level=error msg="failed to receive event from containerd: rpc error: code = 13 desc = \"transport is closing\"" 
INFO[82619] stopping containerd after receiving terminated 
Waiting for /var/run/docker.sock
time="2016-04-20T17:10:49.089498653+02:00" level=info msg="New containerd process, pid: 7209\n" 
/var/run/docker.sock is up
time="2016-04-20T17:10:50.216630765+02:00" level=info msg="[graphdriver] using prior storage driver \"aufs\"" 
time="2016-04-20T17:10:50.233111307+02:00" level=info msg="Graph migration to content-addressability took 0.00 seconds" 
time="2016-04-20T17:10:50.243068742+02:00" level=info msg="Firewalld running: false" 
time="2016-04-20T17:10:50.405077217+02:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address" 
time="2016-04-20T17:10:50.511932235+02:00" level=warning msg="Your kernel does not support swap memory limit." 
time="2016-04-20T17:10:50.512280868+02:00" level=warning msg="mountpoint for pids not found" 
time="2016-04-20T17:10:50.512769455+02:00" level=info msg="Loading containers: start." 
......
time="2016-04-20T17:10:50.728399087+02:00" level=info msg="Loading containers: done." 
time="2016-04-20T17:10:50.728437860+02:00" level=info msg="Daemon has completed initialization" 
time="2016-04-20T17:10:50.728457669+02:00" level=info msg="Docker daemon" commit=4dc5990 graphdriver=aufs version=1.11.0 
time="2016-04-20T17:10:50.736912313+02:00" level=info msg="API listen on /var/run/docker.sock" 
time="2016-04-20T17:11:29.049288646+02:00" level=info msg="Processing signal 'terminated'" 
time="2016-04-20T17:11:29.100538573+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T17:11:29.119976741+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T17:11:29.140200848+02:00" level=error msg="restartmanager canceled" 
time="2016-04-20T17:11:29.272028769+02:00" level=error msg="failed to receive event from containerd: rpc error: code = 13 desc = \"transport is closing\"" 
INFO[0040] stopping containerd after receiving terminated 
Waiting for /var/run/docker.sock
time="2016-04-20T17:11:30.342030315+02:00" level=info msg="New containerd process, pid: 7866\n" 
/var/run/docker.sock is up
time="2016-04-20T17:11:31.460700440+02:00" level=info msg="[graphdriver] using prior storage driver \"aufs\"" 
time="2016-04-20T17:11:31.480180570+02:00" level=info msg="Graph migration to content-addressability took 0.00 seconds" 
time="2016-04-20T17:11:31.489517420+02:00" level=info msg="Firewalld running: false" 
time="2016-04-20T17:11:31.660407379+02:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address" 
time="2016-04-20T17:11:31.785197849+02:00" level=warning msg="Your kernel does not support swap memory limit." 
time="2016-04-20T17:11:31.785696392+02:00" level=warning msg="mountpoint for pids not found" 
time="2016-04-20T17:11:31.786523451+02:00" level=info msg="Loading containers: start." 
.......
time="2016-04-20T17:11:32.021421779+02:00" level=info msg="Loading containers: done." 
time="2016-04-20T17:11:32.021473981+02:00" level=info msg="Daemon has completed initialization" 
time="2016-04-20T17:11:32.021497391+02:00" level=info msg="Docker daemon" commit=4dc5990 graphdriver=aufs version=1.11.0 
time="2016-04-20T17:11:32.034245364+02:00" level=info msg="API listen on /var/run/docker.sock" 
time="2016-04-20T17:14:23.008581084+02:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 1.13, use `=` instead." 
time="2016-04-20T17:14:23.400088743+02:00" level=info msg="remote ipam driver weavemesh does not support capabilities"

Disabling ufw altogether didn't change anything. Should I try uninstalling and reinstalling Docker as well? Thank you for your time.

bboreham commented 8 years ago

can you do ip route inside the container please?

weavebug3256 commented 8 years ago
root@a1:/# ip route
default via 172.18.0.1 dev eth1 
10.32.0.0/12 dev ethwe0  proto kernel  scope link  src 10.32.0.1 
172.18.0.0/16 dev eth1  proto kernel  scope link  src 172.18.0.2 
224.0.0.0/4 dev ethwe0  scope link
bboreham commented 8 years ago

That looks ok; the eth1 interface should be on the docker-gwbridge network which connects to the rest of the world and ethwe0 is on the Weave network.

I'm stumped right now.

If you can do it, deleting everything in the Docker directory (/var/lib/docker usually) and restarting may help.

weavebug3256 commented 8 years ago

Unfortunately deleting everything in the /var/lib/docker directory and a complete reinstall of Docker & Weave did not help as well...

rade commented 8 years ago

I don't want to think about environment variables, whether I do it all with sudo or in a root shell, whether I am in the same shell when I eval $(weave env) and run my docker container, etc.

You can use $(weave config) instead, e.g.

sudo docker run $(weave config) -it --name a1 ubuntu:trusty

or, if you need sudo to run weave...

sudo docker run $(sudo weave config) -it --name a1 ubuntu:trusty
bboreham commented 8 years ago

I don't understand why something like Unable to connect to plugin: <whichever Weave plugin> would occur in the first place.

It looks from your logs like Docker (or the whole machine) restarted, and the plugin was not running at the time Docker tried to talk to it. There is a similar issue at #2222, but the symptom is rather different.

To recap, you have a container which seems to be connected to the Weave network, but has no other connectivity. Having no other connectivity would lead directly to being unable to contact WeaveDNS. Your ip route output suggests a docker_gwbridge connection which is how that connectivity should work.

Can you check that this happens repeatably, i.e. if you run commands like in https://github.com/weaveworks/weave/issues/2191#issuecomment-212378455 you get the same failure?

Could you please do docker network inspect docker_gwbridge and see if the container ID appears as an endpoint?

bboreham commented 8 years ago

@weavebug3256 do you have any more information?

rade commented 8 years ago

Closing due to inability to reproduce. @weavebug3256 if you have further info then please re-open.

lubars commented 7 years ago

I'm having the same issue as @weavebug3256, namely that docker seems to be ignoring the --dns argument when using a custom network:

docker --network=weave --dns=172.17.0.1 --dns-search=weave.local. a1

These are the same values given by weave dns-args, which also match the output of docker inspect -f '{{.HostConfig.DNS}}' a1.

Resulting /etc/resolv.conf:

search weave.local
nameserver 127.0.0.11
options ndots:0

I've read that weave run behaves better in this respect than docker run --network=weave but are deploying remotely (via docker --host) and didn't want to resort to using ssh.

Versions I'm running are weave 1.8.2 and docker (engine) 1.12.5.

UPDATE: I have found that setting dns in the /etc/docker/daemon.json file works. I'm creating the file prior to docker installation to avoid a restart:

{
  "dns": ["172.17.0.1"]
}

Note that /etc/resolv.conf remains unchanged (in both host and container), but the docker daemon knows about it and acts accordingly.

bboreham commented 7 years ago

@lubars Docker do not offer a way to disable this behaviour when using a Docker Network plugin.

You can use the Weave Proxy or weave run instead, then Docker won't do this. There is no need to use ssh; merely point DOCKER_HOST at the target machine.

For example:

$ export DOCKER_HOST=other_host:2735
$ weave run <some-image>

or

$ export DOCKER_HOST=other_host:2735
$ weave launch
$ eval $(weave env)
$ docker run <some-image>

Note I launched Weave (remotely) after setting DOCKER_HOST - see https://www.weave.works/docs/net/latest/weave-docker-api/ for more information about how the proxy listens and its use of TLS.

lubars commented 7 years ago

Thanks for your reply @bboreham. Please see my updated solution above.

Other reasons not to go the weave run route (even before you explained that it follows DOCKER_HOST semantics) is that we allow several networking options (including host) and were hoping to maintain a pure docker client image (deploying the zoo of scripts and technologies on the server side).

bboreham commented 7 years ago

Re your update at https://github.com/weaveworks/weave/issues/2191#issuecomment-270729075 - you seem to be saying that Docker will use that address as a fallback after consulting its own data, which is true. This should also be true if you say docker run --dns ...

lubars commented 7 years ago

This is the very bug/behavior that I am reporting, that docker appears to be completely ignoring the --dns value I'm supplying to docker run (whether explicitly or via $(weave dns-args)). The problem resolved after manually adding nameserver 172.17.0.1 to the host's /etc/resolv.conf file, which led me down the path above (trying the daemon.json approach). I'm happy to provide additional debug information.

bboreham commented 7 years ago

You reported that it didn't show up in /etc/resolv.conf.

Docked interposes its own DNS service at 127.0.0.11, and should then use the one you specify as a fall-back if it cannot find an answer there.

lubars commented 7 years ago

@bboreham, it occurred to me that I was restarting docker with the daemon.json method but not with the --dns method, so I tried unconditionally restarting docker immediately after weave installation, and now the --dns flag appears to be working (and without employing any of the techniques or workarounds above). Is this expected? I don't recall having seen anything about restarting docker in the weave documentation.

bboreham commented 7 years ago

No, I've never seen that happen.

Can you post a transcript, for interest?

lubars commented 7 years ago

FYI, this problem only appears to happen on Amazon EC2 compute instances; the restart isn't required on vSphere. One difference I can think of off the bat is that on EC2 the compute instances are on their own subnet, which manifests as the 10.32.x.x entries in the container's /etc/hosts file:

127.0.0.1       localhost
::1     localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
10.32.0.1       SM-PROD-0001.weave.local
172.18.0.2     SM-PROD-0001.weave.local

@bboreham, I'm happy to provide a transcript, but not sure what that entails - can you be more specific?