search

weaveworks / weave

Simple, resilient multi-host containers networking and more.
https://www.weave.works
Apache License 2.0
6.62k stars 670 forks source link

Adding OVS bridges to Kubernetes worker-nodes break the Pod Network #3822

Open asoloa opened 4 years ago

asoloa commented 4 years ago

I was directed here by the Kubernetes support. See https://github.com/kubernetes/kubernetes/issues/92282

I have set up a multi-node Kubernetes cluster in CentOS. Currently using Weave Net for the pod network.

What happened?

Scenarios:

  1. Before adding OVS bridges to the worker nodes, the network works as expected (eg. pod can communicate with another pod in the other node).
  2. However after adding OVS bridges to the worker nodes (a simple ovs-vsctl add-br br1), the network breaks (eg. pod can't communicate with another pod in the other node, but pods within the same nodes can).:

What you expected to happen?

AFAIK, the networking between pods shouldn't be affected by the addition of the OVS bridges. Or am I understanding this wrong? Could this be a config-related issue? Additional procedures I may have missed?

Any help would be greatly appreciated. Thank you.

How to reproduce it?

  1. Installed containerd, docker, then kubeadm.
  2. kubeadm init
  3. Installed pod network (Weave Net)
  4. Created 2 sample pods (1 each node)
  5. Check pod connection via ping.
  6. Add OVS bridge in the nodes. (ovs-vsctl add-br br1)
  7. Check pod connection via ping. (No connection)

Versions:

Logs:

$ kubectl logs -n kube-system <weave-net-pod> weave
INFO: 2020/06/18 15:41:10.952130 Command line options: map[conn-limit:200 datapath:datapath db-prefix:/weavedb/weave-net docker-api: expect-npc:true host-root:/host http-addr:127.0.0.1:6784 ipalloc-init:consensus=2 ipalloc-range:10.32.0.0/12 metrics-addr:0.0.0.0:6782 name:9e:ce:d2:46:ab:7a nickname:k8s-master no-dns:true port:6783]
INFO: 2020/06/18 15:41:10.952175 weave  2.6.5
INFO: 2020/06/18 15:41:11.090486 Bridge type is bridged_fastdp
INFO: 2020/06/18 15:41:11.090501 Communication between peers is unencrypted.
INFO: 2020/06/18 15:41:11.109495 Our name is 9e:ce:d2:46:ab:7a(k8s-master)
INFO: 2020/06/18 15:41:11.109538 Launch detected - using supplied peer list: [192.168.2.101 192.168.2.102]
INFO: 2020/06/18 15:41:11.120262 Unable to fetch ConfigMap kube-system/weave-net to infer unique cluster ID
INFO: 2020/06/18 15:41:11.120289 Checking for pre-existing addresses on weave bridge
INFO: 2020/06/18 15:41:11.169888 [allocator 9e:ce:d2:46:ab:7a] No valid persisted data
INFO: 2020/06/18 15:41:11.209249 [allocator 9e:ce:d2:46:ab:7a] Initialising via deferred consensus
INFO: 2020/06/18 15:41:11.209315 Sniffing traffic on datapath (via ODP)
INFO: 2020/06/18 15:41:11.214757 Error checking version: Get https://checkpoint-api.weave.works/v1/check/weave-net?arch=amd64&flag_docker-version=none&flag_kernel-version=3.10.0-1127.10.1.el7.x86_64&flag_kubernetes-version=v1.18.4&os=linux&signature=aqJsxMxmynrA0ZARm71xGpfwfQ1CWB6ygLElAmOst4I%3D&version=2.6.5: dial tcp: lookup checkpoint-api.weave.works on 10.96.0.10:53: write udp 192.168.2.100:41373->10.96.0.10:53: write: operation not permitted
INFO: 2020/06/18 15:41:11.214820 ->[192.168.2.101:6783] attempting connection
INFO: 2020/06/18 15:41:11.214944 ->[192.168.2.102:6783] attempting connection
INFO: 2020/06/18 15:41:11.215269 ->[192.168.2.101:6783] error during connection attempt: dial tcp :0->192.168.2.101:6783: connect: connection refused
INFO: 2020/06/18 15:41:11.215332 ->[192.168.2.102:6783] error during connection attempt: dial tcp :0->192.168.2.102:6783: connect: connection refused
INFO: 2020/06/18 15:41:11.219175 Listening for HTTP control messages on 127.0.0.1:6784
INFO: 2020/06/18 15:41:11.219358 Listening for metrics requests on 0.0.0.0:6782
INFO: 2020/06/18 15:41:12.026576 [kube-peers] Added myself to peer list &{[{9e:ce:d2:46:ab:7a k8s-master}]}
DEBU: 2020/06/18 15:41:12.036563 [kube-peers] Nodes that have disappeared: map[]
INFO: 2020/06/18 15:41:12.639696 ->[192.168.2.101:6783] attempting connection
INFO: 2020/06/18 15:41:12.640311 ->[192.168.2.101:6783] error during connection attempt: dial tcp :0->192.168.2.101:6783: connect: connection refused
INFO: 2020/06/18 15:41:13.606887 ->[192.168.2.102:6783] attempting connection
INFO: 2020/06/18 15:41:13.607749 ->[192.168.2.102:6783] error during connection attempt: dial tcp :0->192.168.2.102:6783: connect: connection refused
INFO: 2020/06/18 15:41:16.023489 ->[192.168.2.101:6783] attempting connection
INFO: 2020/06/18 15:41:16.023992 ->[192.168.2.101:6783] error during connection attempt: dial tcp :0->192.168.2.101:6783: connect: connection refused
INFO: 2020/06/18 15:41:16.437482 ->[192.168.2.102:6783] attempting connection
INFO: 2020/06/18 15:41:16.438342 ->[192.168.2.102:6783] error during connection attempt: dial tcp :0->192.168.2.102:6783: connect: connection refused
INFO: 2020/06/18 15:41:22.034982 ->[192.168.2.102:6783] attempting connection
INFO: 2020/06/18 15:41:22.038370 ->[192.168.2.102:6783] error during connection attempt: dial tcp :0->192.168.2.102:6783: connect: connection refused
INFO: 2020/06/18 15:41:22.558678 ->[192.168.2.101:6783] attempting connection
INFO: 2020/06/18 15:41:22.559385 ->[192.168.2.101:6783] error during connection attempt: dial tcp :0->192.168.2.101:6783: connect: connection refused
INFO: 2020/06/18 15:41:31.852632 ->[192.168.2.102:6783] attempting connection
INFO: 2020/06/18 15:41:31.853641 ->[192.168.2.102:6783] error during connection attempt: dial tcp :0->192.168.2.102:6783: connect: connection refused
INFO: 2020/06/18 15:41:32.680477 ->[192.168.2.101:6783] attempting connection
INFO: 2020/06/18 15:41:32.681255 ->[192.168.2.101:6783] error during connection attempt: dial tcp :0->192.168.2.101:6783: connect: connection refused
INFO: 2020/06/18 15:41:37.871326 ->[192.168.2.102:6783] attempting connection
INFO: 2020/06/18 15:41:37.873555 ->[192.168.2.102:6783] error during connection attempt: dial tcp :0->192.168.2.102:6783: connect: connection refused
INFO: 2020/06/18 15:41:45.380284 ->[192.168.2.101:6783] attempting connection
INFO: 2020/06/18 15:41:45.380928 ->[192.168.2.101:6783] error during connection attempt: dial tcp :0->192.168.2.101:6783: connect: connection refused
INFO: 2020/06/18 15:41:53.357396 ->[192.168.2.102:6783] attempting connection
INFO: 2020/06/18 15:41:53.358136 ->[192.168.2.102:6783] error during connection attempt: dial tcp :0->192.168.2.102:6783: connect: connection refused
INFO: 2020/06/18 15:42:01.959041 ->[192.168.2.101:6783] attempting connection
INFO: 2020/06/18 15:42:01.959673 ->[192.168.2.101:6783] error during connection attempt: dial tcp :0->192.168.2.101:6783: connect: connection refused
INFO: 2020/06/18 15:42:07.428284 ->[192.168.2.102:6783] attempting connection
INFO: 2020/06/18 15:42:07.429264 ->[192.168.2.102:6783] error during connection attempt: dial tcp :0->192.168.2.102:6783: connect: connection refused
INFO: 2020/06/18 15:42:24.672355 ->[192.168.2.101:6783] attempting connection
INFO: 2020/06/18 15:42:24.674406 ->[192.168.2.101:6783] error during connection attempt: dial tcp :0->192.168.2.101:6783: connect: connection refused
INFO: 2020/06/18 15:42:43.100919 ->[192.168.2.102:33791] connection accepted
INFO: 2020/06/18 15:42:43.113546 ->[192.168.2.102:33791|f6:46:e0:09:15:6d(k8s-worker-2)]: connection ready; using protocol version 2
INFO: 2020/06/18 15:42:43.113773 overlay_switch ->[f6:46:e0:09:15:6d(k8s-worker-2)] using fastdp
INFO: 2020/06/18 15:42:43.113798 ->[192.168.2.102:33791|f6:46:e0:09:15:6d(k8s-worker-2)]: connection added (new peer)
10.32.0.1
DEBU: 2020/06/18 15:42:43.324303 registering for updates for node delete events
INFO: 2020/06/18 15:42:48.500826 ->[192.168.2.101:33217] connection accepted
INFO: 2020/06/18 15:42:48.508041 ->[192.168.2.101:33217|c6:c1:80:a3:56:1d(k8s-worker-1)]: connection ready; using protocol version 2
INFO: 2020/06/18 15:42:48.508100 overlay_switch ->[c6:c1:80:a3:56:1d(k8s-worker-1)] using fastdp
INFO: 2020/06/18 15:42:48.508266 ->[192.168.2.101:33217|c6:c1:80:a3:56:1d(k8s-worker-1)]: connection added (new peer)
INFO: 2020/06/18 15:43:43.115857 overlay_switch ->[f6:46:e0:09:15:6d(k8s-worker-2)] sleeve timed out waiting for UDP heartbeat
INFO: 2020/06/18 15:43:43.120643 ->[192.168.2.102:33791|f6:46:e0:09:15:6d(k8s-worker-2)]: connection shutting down due to error: no working forwarders to f6:46:e0:09:15:6d(k8s-worker-2)
INFO: 2020/06/18 15:43:43.120711 ->[192.168.2.102:33791|f6:46:e0:09:15:6d(k8s-worker-2)]: connection deleted
INFO: 2020/06/18 15:43:43.120785 ->[192.168.2.102:6783] attempting connection
INFO: 2020/06/18 15:43:43.121193 ->[192.168.2.102:43942] connection accepted
INFO: 2020/06/18 15:43:43.126261 ->[192.168.2.102:43942|f6:46:e0:09:15:6d(k8s-worker-2)]: connection ready; using protocol version 2
INFO: 2020/06/18 15:43:43.126469 overlay_switch ->[f6:46:e0:09:15:6d(k8s-worker-2)] using fastdp
INFO: 2020/06/18 15:43:43.126509 ->[192.168.2.102:43942|f6:46:e0:09:15:6d(k8s-worker-2)]: connection added (new peer)
INFO: 2020/06/18 15:43:43.129342 ->[192.168.2.102:6783|f6:46:e0:09:15:6d(k8s-worker-2)]: connection ready; using protocol version 2
INFO: 2020/06/18 15:43:43.131166 overlay_switch ->[f6:46:e0:09:15:6d(k8s-worker-2)] using fastdp
INFO: 2020/06/18 15:43:43.131202 ->[192.168.2.102:6783|f6:46:e0:09:15:6d(k8s-worker-2)]: connection shutting down due to error: Multiple connections to f6:46:e0:09:15:6d(k8s-worker-2) added to 9e:ce:d2:46:ab:7a(k8s-master)
INFO: 2020/06/18 15:43:48.508451 ->[192.168.2.101:33217|c6:c1:80:a3:56:1d(k8s-worker-1)]: connection shutting down due to error: read tcp 192.168.2.100:6783->192.168.2.101:33217: read: connection reset by peer
INFO: 2020/06/18 15:43:48.508489 ->[192.168.2.101:33217|c6:c1:80:a3:56:1d(k8s-worker-1)]: connection deleted
INFO: 2020/06/18 15:43:48.508577 ->[192.168.2.101:6783] attempting connection
INFO: 2020/06/18 15:43:48.508987 ->[192.168.2.101:55344] connection accepted
INFO: 2020/06/18 15:43:48.510224 ->[192.168.2.101:55344|c6:c1:80:a3:56:1d(k8s-worker-1)]: connection ready; using protocol version 2
INFO: 2020/06/18 15:43:48.510341 overlay_switch ->[c6:c1:80:a3:56:1d(k8s-worker-1)] using fastdp
INFO: 2020/06/18 15:43:48.510364 ->[192.168.2.101:55344|c6:c1:80:a3:56:1d(k8s-worker-1)]: connection added (new peer)
INFO: 2020/06/18 15:43:48.511689 ->[192.168.2.101:6783|c6:c1:80:a3:56:1d(k8s-worker-1)]: connection ready; using protocol version 2
INFO: 2020/06/18 15:43:48.511768 overlay_switch ->[c6:c1:80:a3:56:1d(k8s-worker-1)] using fastdp
INFO: 2020/06/18 15:43:48.511785 ->[192.168.2.101:6783|c6:c1:80:a3:56:1d(k8s-worker-1)]: connection shutting down due to error: Multiple connections to c6:c1:80:a3:56:1d(k8s-worker-1) added to 9e:ce:d2:46:ab:7a(k8s-master)
INFO: 2020/06/18 15:44:43.126523 ->[192.168.2.102:43942|f6:46:e0:09:15:6d(k8s-worker-2)]: connection shutting down due to error: read tcp 192.168.2.100:6783->192.168.2.102:43942: read: connection reset by peer
INFO: 2020/06/18 15:44:43.127089 ->[192.168.2.102:43942|f6:46:e0:09:15:6d(k8s-worker-2)]: connection deleted
INFO: 2020/06/18 15:44:43.127121 ->[192.168.2.102:35210] connection accepted
INFO: 2020/06/18 15:44:43.127775 ->[192.168.2.102:35210|f6:46:e0:09:15:6d(k8s-worker-2)]: connection ready; using protocol version 2
INFO: 2020/06/18 15:44:43.127843 overlay_switch ->[f6:46:e0:09:15:6d(k8s-worker-2)] using fastdp
INFO: 2020/06/18 15:44:43.127855 ->[192.168.2.102:35210|f6:46:e0:09:15:6d(k8s-worker-2)]: connection added (new peer)
INFO: 2020/06/18 15:44:48.510612 overlay_switch ->[c6:c1:80:a3:56:1d(k8s-worker-1)] using sleeve
INFO: 2020/06/18 15:44:48.510684 overlay_switch ->[c6:c1:80:a3:56:1d(k8s-worker-1)] sleeve timed out waiting for UDP heartbeat
INFO: 2020/06/18 15:44:48.510765 ->[192.168.2.101:55344|c6:c1:80:a3:56:1d(k8s-worker-1)]: connection shutting down due to error: no working forwarders to c6:c1:80:a3:56:1d(k8s-worker-1)
INFO: 2020/06/18 15:44:48.510914 ->[192.168.2.101:55344|c6:c1:80:a3:56:1d(k8s-worker-1)]: connection deleted
INFO: 2020/06/18 15:44:48.510987 ->[192.168.2.101:6783] attempting connection
INFO: 2020/06/18 15:44:48.512701 ->[192.168.2.101:6783|c6:c1:80:a3:56:1d(k8s-worker-1)]: connection ready; using protocol version 2
INFO: 2020/06/18 15:44:48.512822 overlay_switch ->[c6:c1:80:a3:56:1d(k8s-worker-1)] using fastdp
INFO: 2020/06/18 15:44:48.512837 ->[192.168.2.101:6783|c6:c1:80:a3:56:1d(k8s-worker-1)]: connection added (new peer)
INFO: 2020/06/18 15:44:48.513898 ->[192.168.2.101:32890] connection accepted
INFO: 2020/06/18 15:44:48.515024 ->[192.168.2.101:32890|c6:c1:80:a3:56:1d(k8s-worker-1)]: connection ready; using protocol version 2
INFO: 2020/06/18 15:44:48.515260 overlay_switch ->[c6:c1:80:a3:56:1d(k8s-worker-1)] using fastdp
INFO: 2020/06/18 15:44:48.515281 ->[192.168.2.101:6783|c6:c1:80:a3:56:1d(k8s-worker-1)]: connection deleted
INFO: 2020/06/18 15:44:48.515293 ->[192.168.2.101:6783|c6:c1:80:a3:56:1d(k8s-worker-1)]: connection shutting down due to error: Multiple connections to c6:c1:80:a3:56:1d(k8s-worker-1) added to 9e:ce:d2:46:ab:7a(k8s-master)
INFO: 2020/06/18 15:44:48.515313 ->[192.168.2.101:32890|c6:c1:80:a3:56:1d(k8s-worker-1)]: connection added (new peer)
INFO: 2020/06/18 15:44:49.018146 overlay_switch ->[c6:c1:80:a3:56:1d(k8s-worker-1)] using sleeve
INFO: 2020/06/18 15:44:49.018175 ->[192.168.2.101:32890|c6:c1:80:a3:56:1d(k8s-worker-1)]: connection fully established
INFO: 2020/06/18 15:44:49.020226 sleeve ->[192.168.2.101:6783|c6:c1:80:a3:56:1d(k8s-worker-1)]: Effective MTU verified at 1438
INFO: 2020/06/18 15:45:43.128925 overlay_switch ->[f6:46:e0:09:15:6d(k8s-worker-2)] using sleeve
INFO: 2020/06/18 15:45:43.129008 overlay_switch ->[f6:46:e0:09:15:6d(k8s-worker-2)] sleeve timed out waiting for UDP heartbeat
INFO: 2020/06/18 15:45:43.129047 ->[192.168.2.102:35210|f6:46:e0:09:15:6d(k8s-worker-2)]: connection shutting down due to error: no working forwarders to f6:46:e0:09:15:6d(k8s-worker-2)
INFO: 2020/06/18 15:45:43.129305 ->[192.168.2.102:35210|f6:46:e0:09:15:6d(k8s-worker-2)]: connection deleted
INFO: 2020/06/18 15:45:43.129596 ->[192.168.2.102:6783] attempting connection
INFO: 2020/06/18 15:45:43.131586 ->[192.168.2.102:45355] connection accepted
INFO: 2020/06/18 15:45:43.133225 ->[192.168.2.102:6783|f6:46:e0:09:15:6d(k8s-worker-2)]: connection ready; using protocol version 2
INFO: 2020/06/18 15:45:43.133259 overlay_switch ->[f6:46:e0:09:15:6d(k8s-worker-2)] using fastdp
INFO: 2020/06/18 15:45:43.133273 ->[192.168.2.102:6783|f6:46:e0:09:15:6d(k8s-worker-2)]: connection added
INFO: 2020/06/18 15:45:43.133621 ->[192.168.2.102:45355|f6:46:e0:09:15:6d(k8s-worker-2)]: connection ready; using protocol version 2
INFO: 2020/06/18 15:45:43.133675 overlay_switch ->[f6:46:e0:09:15:6d(k8s-worker-2)] using fastdp
INFO: 2020/06/18 15:45:43.133718 ->[192.168.2.102:6783|f6:46:e0:09:15:6d(k8s-worker-2)]: connection deleted
INFO: 2020/06/18 15:45:43.133758 ->[192.168.2.102:45355|f6:46:e0:09:15:6d(k8s-worker-2)]: connection added
INFO: 2020/06/18 15:45:43.133839 ->[192.168.2.102:6783|f6:46:e0:09:15:6d(k8s-worker-2)]: connection shutting down due to error: Multiple connections to f6:46:e0:09:15:6d(k8s-worker-2) added to 9e:ce:d2:46:ab:7a(k8s-master)
INFO: 2020/06/18 15:45:43.135522 overlay_switch ->[f6:46:e0:09:15:6d(k8s-worker-2)] using sleeve
INFO: 2020/06/18 15:45:43.135599 ->[192.168.2.102:45355|f6:46:e0:09:15:6d(k8s-worker-2)]: connection fully established
INFO: 2020/06/18 15:45:43.135937 sleeve ->[192.168.2.102:6783|f6:46:e0:09:15:6d(k8s-worker-2)]: Effective MTU verified at 1438
INFO: 2020/06/18 15:48:53.896259 Discovered remote MAC c2:56:a6:4a:8d:6e at c6:c1:80:a3:56:1d(k8s-worker-1)
INFO: 2020/06/18 15:48:56.293224 Discovered remote MAC 96:8c:7f:80:e4:1c at f6:46:e0:09:15:6d(k8s-worker-2)
INFO: 2020/06/18 15:49:53.114589 Discovered remote MAC 4e:d1:7d:18:d1:14 at c6:c1:80:a3:56:1d(k8s-worker-1)
netlink reply sequence number mismatch (got 1083, expected 1084)
WARN: 2020/06/18 15:50:42.478392 netlink error response: no such device
WARN: 2020/06/18 15:50:42.478411 netlink error response: no such device
netlink reply sequence number mismatch (got 1086, expected 1088)
netlink reply sequence number mismatch (got 1087, expected 1088)
WARN: 2020/06/18 15:50:52.474032 netlink error response: no such device
WARN: 2020/06/18 15:50:52.474046 netlink error response: no such device
netlink reply sequence number mismatch (got 1090, expected 1092)
netlink reply sequence number mismatch (got 1091, expected 1092)
WARN: 2020/06/18 15:51:02.473697 netlink error response: no such device
WARN: 2020/06/18 15:51:02.473727 netlink error response: no such device
WARN: 2020/06/18 15:51:11.110094 netlink error response: no such device
netlink reply sequence number mismatch (got 1094, expected 1096)
netlink reply sequence number mismatch (got 1095, expected 1096)
WARN: 2020/06/18 15:51:12.474542 netlink error response: no such device
WARN: 2020/06/18 15:51:12.474608 netlink error response: no such device
WARN: 2020/06/18 15:51:22.474820 netlink error response: no such device
WARN: 2020/06/18 15:51:22.474867 netlink error response: no such device
netlink reply sequence number mismatch (got 1099, expected 1101)
netlink reply sequence number mismatch (got 1100, expected 1101)
WARN: 2020/06/18 15:51:32.473958 netlink error response: no such device
WARN: 2020/06/18 15:51:32.473985 netlink error response: no such device
netlink reply sequence number mismatch (got 1103, expected 1105)
netlink reply sequence number mismatch (got 1104, expected 1105)
WARN: 2020/06/18 15:51:42.473594 netlink error response: no such device
WARN: 2020/06/18 15:51:42.473611 netlink error response: no such device
netlink reply sequence number mismatch (got 1107, expected 1109)
netlink reply sequence number mismatch (got 1108, expected 1109)
WARN: 2020/06/18 15:51:52.473293 netlink error response: no such device
WARN: 2020/06/18 15:51:52.473308 netlink error response: no such device
netlink reply sequence number mismatch (got 1111, expected 1113)
netlink reply sequence number mismatch (got 1112, expected 1113)
netlink reply sequence number mismatch (got 1115, expected 1117)
netlink reply sequence number mismatch (got 1116, expected 1117)
WARN: 2020/06/18 15:52:02.475008 netlink error response: no such device
WARN: 2020/06/18 15:52:02.475061 netlink error response: no such device
netlink reply sequence number mismatch (got 1119, expected 1121)
netlink reply sequence number mismatch (got 1120, expected 1121)
WARN: 2020/06/18 15:52:12.474521 netlink error response: no such device
WARN: 2020/06/18 15:52:12.474544 netlink error response: no such device
WARN: 2020/06/18 15:52:22.473488 netlink error response: no such device
WARN: 2020/06/18 15:52:22.473505 netlink error response: no such device
netlink reply sequence number mismatch (got 1123, expected 1125)
netlink reply sequence number mismatch (got 1124, expected 1125)
netlink reply sequence number mismatch (got 1127, expected 1129)
netlink reply sequence number mismatch (got 1128, expected 1129)
WARN: 2020/06/18 15:52:32.475453 netlink error response: no such device
WARN: 2020/06/18 15:52:32.475497 netlink error response: no such device
netlink reply sequence number mismatch (got 1131, expected 1133)
netlink reply sequence number mismatch (got 1132, expected 1133)
WARN: 2020/06/18 15:52:42.473192 netlink error response: no such device
WARN: 2020/06/18 15:52:42.473211 netlink error response: no such device
netlink reply sequence number mismatch (got 1135, expected 1137)
netlink reply sequence number mismatch (got 1136, expected 1137)
WARN: 2020/06/18 15:52:52.475174 netlink error response: no such device
WARN: 2020/06/18 15:52:52.475191 netlink error response: no such device
netlink reply sequence number mismatch (got 1139, expected 1141)
netlink reply sequence number mismatch (got 1140, expected 1141)
WARN: 2020/06/18 15:53:02.473802 netlink error response: no such device
WARN: 2020/06/18 15:53:02.473817 netlink error response: no such device
netlink reply sequence number mismatch (got 1143, expected 1145)
netlink reply sequence number mismatch (got 1144, expected 1145)
WARN: 2020/06/18 15:53:12.473904 netlink error response: no such device
WARN: 2020/06/18 15:53:12.473923 netlink error response: no such device
netlink reply sequence number mismatch (got 1147, expected 1149)
netlink reply sequence number mismatch (got 1148, expected 1149)
WARN: 2020/06/18 15:53:22.473385 netlink error response: no such device
WARN: 2020/06/18 15:53:22.473400 netlink error response: no such device
netlink reply sequence number mismatch (got 1151, expected 1153)
netlink reply sequence number mismatch (got 1152, expected 1153)
WARN: 2020/06/18 15:53:32.474838 netlink error response: no such device
WARN: 2020/06/18 15:53:32.474864 netlink error response: no such device
netlink reply sequence number mismatch (got 1155, expected 1157)
netlink reply sequence number mismatch (got 1156, expected 1157)
WARN: 2020/06/18 15:53:42.474695 netlink error response: no such device
WARN: 2020/06/18 15:53:42.474724 netlink error response: no such device
netlink reply sequence number mismatch (got 1159, expected 1161)
netlink reply sequence number mismatch (got 1160, expected 1161)
WARN: 2020/06/18 15:53:52.478891 netlink error response: no such device
WARN: 2020/06/18 15:53:52.478917 netlink error response: no such device
netlink reply sequence number mismatch (got 1163, expected 1165)
netlink reply sequence number mismatch (got 1164, expected 1165)
WARN: 2020/06/18 15:54:02.475085 netlink error response: no such device
WARN: 2020/06/18 15:54:02.475146 netlink error response: no such device
netlink reply sequence number mismatch (got 1167, expected 1169)
netlink reply sequence number mismatch (got 1168, expected 1169)
WARN: 2020/06/18 15:54:12.475078 netlink error response: no such device
WARN: 2020/06/18 15:54:12.475212 netlink error response: no such device
netlink reply sequence number mismatch (got 1171, expected 1173)
netlink reply sequence number mismatch (got 1172, expected 1173)
WARN: 2020/06/18 15:54:22.473385 netlink error response: no such device
WARN: 2020/06/18 15:54:22.473475 netlink error response: no such device
netlink reply sequence number mismatch (got 1175, expected 1177)
netlink reply sequence number mismatch (got 1176, expected 1177)
WARN: 2020/06/18 15:54:32.474121 netlink error response: no such device
WARN: 2020/06/18 15:54:32.474153 netlink error response: no such device
netlink reply sequence number mismatch (got 1179, expected 1181)
netlink reply sequence number mismatch (got 1180, expected 1181)
WARN: 2020/06/18 15:54:42.475395 netlink error response: no such device
WARN: 2020/06/18 15:54:42.475448 netlink error response: no such device
netlink reply sequence number mismatch (got 1183, expected 1185)
netlink reply sequence number mismatch (got 1184, expected 1185)
WARN: 2020/06/18 15:54:52.476812 netlink error response: no such device
WARN: 2020/06/18 15:54:52.476860 netlink error response: no such device
netlink reply sequence number mismatch (got 1187, expected 1189)
netlink reply sequence number mismatch (got 1188, expected 1189)
WARN: 2020/06/18 15:55:02.473179 netlink error response: no such device
WARN: 2020/06/18 15:55:02.473193 netlink error response: no such device
netlink reply sequence number mismatch (got 1191, expected 1193)
netlink reply sequence number mismatch (got 1192, expected 1193)
WARN: 2020/06/18 15:55:12.474066 netlink error response: no such device
WARN: 2020/06/18 15:55:12.474082 netlink error response: no such device
netlink reply sequence number mismatch (got 1195, expected 1197)
netlink reply sequence number mismatch (got 1196, expected 1197)
WARN: 2020/06/18 15:55:22.474118 netlink error response: no such device
WARN: 2020/06/18 15:55:22.474164 netlink error response: no such device
netlink reply sequence number mismatch (got 1199, expected 1201)
netlink reply sequence number mismatch (got 1200, expected 1201)
WARN: 2020/06/18 15:55:32.473236 netlink error response: no such device
WARN: 2020/06/18 15:55:32.473265 netlink error response: no such device
netlink reply sequence number mismatch (got 1203, expected 1205)
netlink reply sequence number mismatch (got 1204, expected 1205)
WARN: 2020/06/18 15:55:42.473597 netlink error response: no such device
WARN: 2020/06/18 15:55:42.473630 netlink error response: no such device
netlink reply sequence number mismatch (got 1207, expected 1209)
netlink reply sequence number mismatch (got 1208, expected 1209)
WARN: 2020/06/18 15:55:52.473337 netlink error response: no such device
WARN: 2020/06/18 15:55:52.473355 netlink error response: no such device
netlink reply sequence number mismatch (got 1211, expected 1213)
netlink reply sequence number mismatch (got 1212, expected 1213)
WARN: 2020/06/18 15:56:02.473489 netlink error response: no such device
WARN: 2020/06/18 15:56:02.473516 netlink error response: no such device
netlink reply sequence number mismatch (got 1215, expected 1217)
netlink reply sequence number mismatch (got 1216, expected 1217)
WARN: 2020/06/18 15:56:11.110218 netlink error response: no such device
WARN: 2020/06/18 15:56:12.474665 netlink error response: no such device
WARN: 2020/06/18 15:56:12.474945 netlink error response: no such device
netlink reply sequence number mismatch (got 1220, expected 1222)
netlink reply sequence number mismatch (got 1221, expected 1222)
WARN: 2020/06/18 15:56:22.476418 netlink error response: no such device
WARN: 2020/06/18 15:56:22.476452 netlink error response: no such device
netlink reply sequence number mismatch (got 1224, expected 1226)
netlink reply sequence number mismatch (got 1225, expected 1226)
WARN: 2020/06/18 15:56:32.473685 netlink error response: no such device
WARN: 2020/06/18 15:56:32.473717 netlink error response: no such device
netlink reply sequence number mismatch (got 1228, expected 1230)
netlink reply sequence number mismatch (got 1229, expected 1230)
WARN: 2020/06/18 15:56:42.474293 netlink error response: no such device
WARN: 2020/06/18 15:56:42.474312 netlink error response: no such device
netlink reply sequence number mismatch (got 1232, expected 1234)
netlink reply sequence number mismatch (got 1233, expected 1234)
WARN: 2020/06/18 15:56:52.474821 netlink error response: no such device
WARN: 2020/06/18 15:56:52.474835 netlink error response: no such device
netlink reply sequence number mismatch (got 1236, expected 1238)
netlink reply sequence number mismatch (got 1237, expected 1238)
WARN: 2020/06/18 15:57:02.474171 netlink error response: no such device
WARN: 2020/06/18 15:57:02.474205 netlink error response: no such device
WARN: 2020/06/18 15:57:12.475287 netlink error response: no such device
WARN: 2020/06/18 15:57:12.475334 netlink error response: no such device
netlink reply sequence number mismatch (got 1240, expected 1242)
netlink reply sequence number mismatch (got 1241, expected 1242)
netlink reply sequence number mismatch (got 1244, expected 1246)
netlink reply sequence number mismatch (got 1245, expected 1246)
WARN: 2020/06/18 15:57:22.473803 netlink error response: no such device
WARN: 2020/06/18 15:57:22.473827 netlink error response: no such device
netlink reply sequence number mismatch (got 1248, expected 1250)
netlink reply sequence number mismatch (got 1249, expected 1250)
WARN: 2020/06/18 15:57:32.473535 netlink error response: no such device
WARN: 2020/06/18 15:57:32.473555 netlink error response: no such device
netlink reply sequence number mismatch (got 1252, expected 1254)
netlink reply sequence number mismatch (got 1253, expected 1254)
WARN: 2020/06/18 15:57:42.475687 netlink error response: no such device
WARN: 2020/06/18 15:57:42.475700 netlink error response: no such device
netlink reply sequence number mismatch (got 1256, expected 1258)
netlink reply sequence number mismatch (got 1257, expected 1258)
WARN: 2020/06/18 15:57:52.475225 netlink error response: no such device
WARN: 2020/06/18 15:57:52.475271 netlink error response: no such device
WARN: 2020/06/18 15:58:02.473574 netlink error response: no such device
WARN: 2020/06/18 15:58:02.473589 netlink error response: no such device
netlink reply sequence number mismatch (got 1260, expected 1262)
netlink reply sequence number mismatch (got 1261, expected 1262)
WARN: 2020/06/18 15:58:12.475094 netlink error response: no such device
WARN: 2020/06/18 15:58:12.475142 netlink error response: no such device
netlink reply sequence number mismatch (got 1264, expected 1266)
netlink reply sequence number mismatch (got 1265, expected 1266)
netlink reply sequence number mismatch (got 1268, expected 1270)
netlink reply sequence number mismatch (got 1269, expected 1270)
WARN: 2020/06/18 15:58:22.475081 netlink error response: no such device
WARN: 2020/06/18 15:58:22.475098 netlink error response: no such device
netlink reply sequence number mismatch (got 1272, expected 1274)
netlink reply sequence number mismatch (got 1273, expected 1274)
WARN: 2020/06/18 15:58:32.474283 netlink error response: no such device
WARN: 2020/06/18 15:58:32.474330 netlink error response: no such device
netlink reply sequence number mismatch (got 1276, expected 1278)
netlink reply sequence number mismatch (got 1277, expected 1278)
WARN: 2020/06/18 15:58:42.474253 netlink error response: no such device
WARN: 2020/06/18 15:58:42.474287 netlink error response: no such device
netlink reply sequence number mismatch (got 1280, expected 1282)
netlink reply sequence number mismatch (got 1281, expected 1282)
WARN: 2020/06/18 15:58:52.473877 netlink error response: no such device
WARN: 2020/06/18 15:58:52.473899 netlink error response: no such device
WARN: 2020/06/18 15:59:02.474616 netlink error response: no such device
netlink reply sequence number mismatch (got 1284, expected 1286)
netlink reply sequence number mismatch (got 1285, expected 1286)
WARN: 2020/06/18 15:59:02.474787 netlink error response: no such device
WARN: 2020/06/18 15:59:12.474449 netlink error response: no such device
WARN: 2020/06/18 15:59:12.474601 netlink error response: no such device
netlink reply sequence number mismatch (got 1288, expected 1290)
netlink reply sequence number mismatch (got 1289, expected 1290)
netlink reply sequence number mismatch (got 1292, expected 1294)
netlink reply sequence number mismatch (got 1293, expected 1294)
WARN: 2020/06/18 15:59:22.474357 netlink error response: no such device
WARN: 2020/06/18 15:59:22.474403 netlink error response: no such device
netlink reply sequence number mismatch (got 1296, expected 1298)
$ weave status peers
f6:46:e0:09:15:6d(k8s-worker-2)
   -> 192.168.2.100:6783    9e:ce:d2:46:ab:7a(k8s-master)         established
   <- 192.168.2.101:38183   c6:c1:80:a3:56:1d(k8s-worker-1)       established
c6:c1:80:a3:56:1d(k8s-worker-1)
   -> 192.168.2.100:6783    9e:ce:d2:46:ab:7a(k8s-master)         established
   -> 192.168.2.102:6783    f6:46:e0:09:15:6d(k8s-worker-2)       established
9e:ce:d2:46:ab:7a(k8s-master)
   <- 192.168.2.102:45355   f6:46:e0:09:15:6d(k8s-worker-2)       established
   <- 192.168.2.101:54482   c6:c1:80:a3:56:1d(k8s-worker-1)       established
bboreham commented 4 years ago

...

Please don't do this; we ask for the logs because there is information there that helps to speed troubleshooting. Please include the whole log, at least the first thousand lines.

asoloa commented 4 years ago

...

Please don't do this; we ask for the logs because there is information there that helps to speed troubleshooting. Please include the whole log, at least the first thousand lines.

@bboreham Apologies. I have updated the logs. Hope this helps. Thank you.

mel1nn commented 2 years ago

Same problem for me.