paleksic commented 5 years ago

I'm seeing bunch of these error messages. Is it safe to ignore them as I've read somewhere it can be logging issue.


INFO: 2019/02/02 22:04:18.894446 Discovered remote MAC 6e:a6:6a:81:53:f3 at 6e:a6:6a:81:53:f3(g-k8s-worker-16)
INFO: 2019/02/02 22:04:18.896455 overlay_switch ->[6e:a6:6a:81:53:f3(g-k8s-worker-16)] using fastdp
INFO: 2019/02/02 22:04:18.897431 Discovered remote MAC 9a:33:5d:f4:36:ed at de:9d:dc:93:55:1a(g-k8s-worker-18)
ERRO: 2019/02/02 22:04:18.897459 Captured frame from MAC (9a:33:5d:f4:36:ed) to (f6:5d:d1:0b:e9:2e) associated with another peer de:9d:dc:93:55:1a(g-k8s-worker-18)
INFO: 2019/02/02 22:04:18.928688 Discovered remote MAC a6:32:42:ae:f6:af at 6e:a6:6a:81:53:f3(g-k8s-worker-16)
ERRO: 2019/02/02 22:04:18.928744 Captured frame from MAC (a6:32:42:ae:f6:af) to (d2:37:59:e0:a5:0e) associated with another peer 6e:a6:6a:81:53:f3(g-k8s-worker-16)
INFO: 2019/02/02 22:04:18.949262 fastdp ->[10.10.40.148:6784|de:9d:dc:93:55:1a(g-k8s-worker-18)]: IPSec init SA remote
INFO: 2019/02/02 22:04:18.949307 ipsec: InitSARemote: 10.10.40.138 -> 10.10.40.148 :6784 0x53a94bc3
INFO: 2019/02/02 22:04:18.952053 overlay_switch ->[de:9d:dc:93:55:1a(g-k8s-worker-18)] using sleeve
INFO: 2019/02/02 22:04:18.952102 ->[10.10.40.148:57491|de:9d:dc:93:55:1a(g-k8s-worker-18)]: connection fully established
INFO: 2019/02/02 22:04:18.952698 EMSGSIZE on send, expecting PMTU update (IP packet was 60052 bytes, payload was 60044 bytes)
INFO: 2019/02/02 22:04:18.955994 sleeve ->[10.10.40.148:6783|de:9d:dc:93:55:1a(g-k8s-worker-18)]: Effective MTU verified at 1414
INFO: 2019/02/02 22:04:18.972268 Discovered remote MAC fa:e7:69:24:51:21 at 6e:a6:6a:81:53:f3(g-k8s-worker-16)
ERRO: 2019/02/02 22:04:18.972326 Captured frame from MAC (fa:e7:69:24:51:21) to (da:f5:6f:0f:c2:74) associated with another peer 6e:a6:6a:81:53:f3(g-k8s-worker-16)
ERRO: 2019/02/02 22:04:18.976111 Captured frame from MAC (fa:e7:69:24:51:21) to (da:f5:6f:0f:c2:74) associated with another peer 6e:a6:6a:81:53:f3(g-k8s-worker-16)
ERRO: 2019/02/02 22:04:19.012230 Captured frame from MAC (d6:43:69:9b:73:48) to (da:f5:6f:0f:c2:74) associated with another peer 02:ba:92:92:a4:b8(g-k8s-worker-11)
INFO: 2019/02/02 22:04:19.057986 overlay_switch ->[de:9d:dc:93:55:1a(g-k8s-worker-18)] using fastdp
INFO: 2019/02/02 22:04:19.060529 Discovered remote MAC da:f5:6f:0f:c2:74 at de:9d:dc:93:55:1a(g-k8s-worker-18)
ERRO: 2019/02/02 22:04:19.060565 Captured frame from MAC (da:f5:6f:0f:c2:74) to (fa:e7:69:24:51:21) associated with another peer de:9d:dc:93:55:1a(g-k8s-worker-18)
INFO: 2019/02/02 22:04:19.095087 Discovered remote MAC de:9d:dc:93:55:1a at de:9d:dc:93:55:1a(g-k8s-worker-18)
ERRO: 2019/02/02 22:04:19.197197 Captured frame from MAC (4e:46:93:1f:4f:24) to (d2:37:59:e0:a5:0e) associated with another peer 02:5a:54:30:fa:53(g-k8s-worker-15)
ERRO: 2019/02/02 22:04:19.219467 Captured frame from MAC (42:31:7e:04:eb:8e) to (d2:37:59:e0:a5:0e) associated with another peer 6a:52:e9:ed:5e:39(g-k8s-worker-13)
INFO: 2019/02/02 22:04:19.265401 Discovered remote MAC fe:2f:ef:30:d1:45 at de:9d:dc:93:55:1a(g-k8s-worker-18)
ERRO: 2019/02/02 22:04:19.395734 Captured frame from MAC (d6:43:69:9b:73:48) to (72:13:8e:d7:9d:ab) associated with another peer 02:ba:92:92:a4:b8(g-k8s-worker-11)
INFO: 2019/02/02 22:04:19.412189 Discovered remote MAC 92:d8:a3:72:3b:ed at 6e:a6:6a:81:53:f3(g-k8s-worker-16)
ERRO: 2019/02/02 22:04:19.420431 Captured frame from MAC (5a:41:f9:da:81:2d) to (d2:37:59:e0:a5:0e) associated with another peer ca:1b:32:f4:6b:d4(g-k8s-worker-10)
INFO: 2019/02/02 22:04:19.454131 Discovered remote MAC 0e:20:2d:ac:c9:d0 at 6e:a6:6a:81:53:f3(g-k8s-worker-16)
INFO: 2019/02/02 22:04:19.693693 Discovered remote MAC f6:5d:d1:0b:e9:2e at 6e:a6:6a:81:53:f3(g-k8s-worker-16)
INFO: 2019/02/02 22:04:19.747192 ->[10.10.40.122:57079] connection accepted
INFO: 2019/02/02 22:04:19.760751 ->[10.10.40.122:57079|36:32:51:ad:16:c5(g-k8s-master-2)]: connection ready; using protocol version 2
INFO: 2019/02/02 22:04:19.760879 overlay_switch ->[36:32:51:ad:16:c5(g-k8s-master-2)] using fastdp
INFO: 2019/02/02 22:04:19.760910 ->[10.10.40.122:57079|36:32:51:ad:16:c5(g-k8s-master-2)]: connection added (new peer)
INFO: 2019/02/02 22:04:19.761228 Setting up IPsec between 92:5a:00:16:48:ea(g-k8s-worker-8) and 36:32:51:ad:16:c5(g-k8s-master-2)
INFO: 2019/02/02 22:04:19.761395 ipsec: InitSALocal: 10.10.40.122 -> 10.10.40.138 :6784 0x6b2bc1b6
INFO: 2019/02/02 22:04:19.867267 fastdp ->[10.10.40.122:6784|36:32:51:ad:16:c5(g-k8s-master-2)]: IPSec init SA remote
INFO: 2019/02/02 22:04:19.867319 ipsec: InitSARemote: 10.10.40.138 -> 10.10.40.122 :6784 0xbf7dab5c
INFO: 2019/02/02 22:04:19.973925 EMSGSIZE on send, expecting PMTU update (IP packet was 60052 bytes, payload was 60044 bytes)
INFO: 2019/02/02 22:04:19.974059 overlay_switch ->[36:32:51:ad:16:c5(g-k8s-master-2)] using sleeve
INFO: 2019/02/02 22:04:19.974088 ->[10.10.40.122:57079|36:32:51:ad:16:c5(g-k8s-master-2)]: connection fully established
INFO: 2019/02/02 22:04:19.979476 overlay_switch ->[36:32:51:ad:16:c5(g-k8s-master-2)] using fastdp
INFO: 2019/02/02 22:04:19.984629 sleeve ->[10.10.40.122:6783|36:32:51:ad:16:c5(g-k8s-master-2)]: Effective MTU verified at 1414
INFO: 2019/02/02 22:04:19.985131 Discovered remote MAC 8e:99:56:03:01:e4 at 36:32:51:ad:16:c5(g-k8s-master-2)
INFO: 2019/02/02 22:04:20.042064 Discovered remote MAC 36:32:51:ad:16:c5 at 36:32:51:ad:16:c5(g-k8s-master-2)
ERRO: 2019/02/02 22:04:20.179906 Captured frame from MAC (da:a8:5f:11:dc:c8) to (72:13:8e:d7:9d:ab) associated with another peer ce:c8:b6:c0:20:70(g-k8s-worker-0)
ERRO: 2019/02/02 22:04:20.273217 Captured frame from MAC (ea:0a:f4:40:d9:70) to (72:13:8e:d7:9d:ab) associated with another peer 4a:ba:2d:d5:2d:cd(g-k8s-worker-17)
INFO: 2019/02/02 22:04:20.278664 Discovered remote MAC 7a:7c:06:d4:45:f8 at 36:32:51:ad:16:c5(g-k8s-master-2)
INFO: 2019/02/02 22:04:20.710878 Discovered remote MAC 72:13:8e:d7:9d:ab at 36:32:51:ad:16:c5(g-k8s-master-2)
INFO: 2019/02/02 22:04:21.151650 Discovered remote MAC d2:37:59:e0:a5:0e at de:9d:dc:93:55:1a(g-k8s-worker-18)
ERRO: 2019/02/02 22:04:21.151709 Captured frame from MAC (d2:37:59:e0:a5:0e) to (4a:61:f6:c3:a7:d6) associated with another peer de:9d:dc:93:55:1a(g-k8s-worker-18)
ERRO: 2019/02/02 22:04:21.152339 Captured frame from MAC (d2:37:59:e0:a5:0e) to (a2:0d:24:f4:e9:06) associated with another peer de:9d:dc:93:55:1a(g-k8s-worker-18)
INFO: 2019/02/02 22:04:21.943794 Discovered remote MAC 26:56:6b:4f:fb:f5 at 6e:a6:6a:81:53:f3(g-k8s-worker-16)
INFO: 2019/02/02 22:04:23.301770 ->[10.10.40.120:6783] attempting connection
INFO: 2019/02/02 22:04:23.302797 ->[10.10.40.120:6783] error during connection attempt: dial tcp4 :0->10.10.40.120:6783: connect: connection refused
INFO: 2019/02/02 22:04:25.957836 Discovered remote MAC d6:5b:35:90:19:eb at de:9d:dc:93:55:1a(g-k8s-worker-18)
INFO: 2019/02/02 22:04:30.678270 ->[10.10.40.120:44970] connection accepted
INFO: 2019/02/02 22:04:30.701265 ->[10.10.40.120:44970|de:1e:9e:8d:5d:19(g-k8s-master-0)]: connection ready; using protocol version 2
INFO: 2019/02/02 22:04:30.701547 overlay_switch ->[de:1e:9e:8d:5d:19(g-k8s-master-0)] using fastdp
INFO: 2019/02/02 22:04:30.703480 ->[10.10.40.120:44970|de:1e:9e:8d:5d:19(g-k8s-master-0)]: connection added (new peer)
INFO: 2019/02/02 22:04:30.717197 Setting up IPsec between 92:5a:00:16:48:ea(g-k8s-worker-8) and de:1e:9e:8d:5d:19(g-k8s-master-0)
INFO: 2019/02/02 22:04:30.717329 ipsec: InitSALocal: 10.10.40.120 -> 10.10.40.138 :6784 0xd23b0040
INFO: 2019/02/02 22:04:30.870891 fastdp ->[10.10.40.120:6784|de:1e:9e:8d:5d:19(g-k8s-master-0)]: IPSec init SA remote
INFO: 2019/02/02 22:04:30.870926 ipsec: InitSARemote: 10.10.40.138 -> 10.10.40.120 :6784 0xbaef27a9
INFO: 2019/02/02 22:04:30.874359 EMSGSIZE on send, expecting PMTU update (IP packet was 60052 bytes, payload was 60044 bytes)
INFO: 2019/02/02 22:04:30.874452 overlay_switch ->[de:1e:9e:8d:5d:19(g-k8s-master-0)] using sleeve
INFO: 2019/02/02 22:04:30.874502 ->[10.10.40.120:44970|de:1e:9e:8d:5d:19(g-k8s-master-0)]: connection fully established
INFO: 2019/02/02 22:04:30.875206 sleeve ->[10.10.40.120:6783|de:1e:9e:8d:5d:19(g-k8s-master-0)]: Effective MTU verified at 1414
INFO: 2019/02/02 22:04:31.000189 overlay_switch ->[de:1e:9e:8d:5d:19(g-k8s-master-0)] using fastdp
INFO: 2019/02/02 22:04:31.050222 Discovered remote MAC 0e:9b:83:a6:dc:b6 at de:1e:9e:8d:5d:19(g-k8s-master-0)
INFO: 2019/02/02 22:04:31.522023 Discovered remote MAC de:1e:9e:8d:5d:19 at de:1e:9e:8d:5d:19(g-k8s-master-0)
INFO: 2019/02/02 22:04:32.134477 Discovered remote MAC 5a:19:5f:1e:a6:20 at 36:32:51:ad:16:c5(g-k8s-master-2)
INFO: 2019/02/02 22:04:32.534986 Discovered remote MAC da:2a:66:72:3d:51 at 6e:a6:6a:81:53:f3(g-k8s-worker-16)
INFO: 2019/02/02 22:04:38.110785 Discovered remote MAC 0a:38:95:28:d0:99 at de:1e:9e:8d:5d:19(g-k8s-master-0)
INFO: 2019/02/02 22:04:39.038824 Discovered remote MAC 72:07:34:51:ac:51 at b6:5d:20:e7:e6:ef(g-k8s-worker-5)
INFO: 2019/02/02 22:08:18.365190 Discovered remote MAC 8a:4b:bb:f5:40:06 at 56:98:7e:b7:2f:ef(g-k8s-worker-12)
INFO: 2019/02/02 22:10:54.929479 Discovered remote MAC 22:7f:69:a1:26:ec at 16:b4:0a:ae:5d:9f(g-k8s-worker-2)
INFO: 2019/02/02 22:17:06.385443 Discovered remote MAC b6:68:7d:e1:a3:29 at 22:72:12:a3:73:c6(g-k8s-worker-7)
INFO: 2019/02/02 22:18:24.601108 Discovered remote MAC 42:32:a9:cc:1a:fd at 6e:a6:6a:81:53:f3(g-k8s-worker-16)
ERRO: 2019/02/02 22:18:28.095558 Captured frame from MAC (22:61:62:5f:62:67) to (72:13:8e:d7:9d:ab) associated with another peer ba:fb:29:f8:cb:84(g-k8s-worker-4)
ERRO: 2019/02/02 22:23:23.611450 Captured frame from MAC (22:61:62:5f:62:67) to (72:13:8e:d7:9d:ab) associated with another peer ba:fb:29:f8:cb:84(g-k8s-worker-4)
INFO: 2019/02/02 22:33:54.532697 Discovered remote MAC 5a:19:5f:1e:a6:20 at 36:32:51:ad:16:c5(g-k8s-master-2)
INFO: 2019/02/02 22:33:59.366384 Discovered remote MAC ba:37:b3:a0:5e:7f at 92:bd:61:89:29:91(g-k8s-worker-6)
ERRO: 2019/02/02 22:33:59.366432 Captured frame from MAC (ba:37:b3:a0:5e:7f) to (26:c3:71:02:3f:e6) associated with another peer 92:bd:61:89:29:91(g-k8s-worker-6)
ERRO: 2019/02/02 22:33:59.369517 Captured frame from MAC (ba:37:b3:a0:5e:7f) to (fa:e7:69:24:51:21) associated with another peer 92:bd:61:89:29:91(g-k8s-worker-6)
INFO: 2019/02/02 22:33:59.532304 Discovered remote MAC 62:76:77:3d:f2:58 at 46:e5:0b:44:b7:67(g-k8s-worker-20)
ERRO: 2019/02/02 22:33:59.532373 Captured frame from MAC (62:76:77:3d:f2:58) to (e2:01:29:ac:0e:87) associated with another peer 46:e5:0b:44:b7:67(g-k8s-worker-20)
ERRO: 2019/02/02 22:33:59.532632 Captured frame from MAC (62:76:77:3d:f2:58) to (e2:01:29:ac:0e:87) associated with another peer 46:e5:0b:44:b7:67(g-k8s-worker-20)
INFO: 2019/02/02 22:33:59.912596 Discovered remote MAC 1a:87:ed:8f:de:63 at 02:5a:54:30:fa:53(g-k8s-worker-15)
ERRO: 2019/02/02 22:33:59.912664 Captured frame from MAC (1a:87:ed:8f:de:63) to (26:c3:71:02:3f:e6) associated with another peer 02:5a:54:30:fa:53(g-k8s-worker-15)
INFO: 2019/02/02 22:34:00.191135 Discovered remote MAC ce:81:a6:be:65:b3 at ca:1b:32:f4:6b:d4(g-k8s-worker-10)
ERRO: 2019/02/02 22:34:00.191180 Captured frame from MAC (ce:81:a6:be:65:b3) to (e2:01:29:ac:0e:87) associated with another peer ca:1b:32:f4:6b:d4(g-k8s-worker-10)
INFO: 2019/02/02 22:34:00.231676 Discovered remote MAC 72:07:34:51:ac:51 at b6:5d:20:e7:e6:ef(g-k8s-worker-5)
ERRO: 2019/02/02 22:34:00.231725 Captured frame from MAC (72:07:34:51:ac:51) to (e2:01:29:ac:0e:87) associated with another peer b6:5d:20:e7:e6:ef(g-k8s-worker-5)
ERRO: 2019/02/02 22:34:00.231811 Captured frame from MAC (72:07:34:51:ac:51) to (e2:01:29:ac:0e:87) associated with another peer b6:5d:20:e7:e6:ef(g-k8s-worker-5)
ERRO: 2019/02/02 22:34:00.231872 Captured frame from MAC (72:07:34:51:ac:51) to (e2:01:29:ac:0e:87) associated with another peer b6:5d:20:e7:e6:ef(g-k8s-worker-5)
ERRO: 2019/02/02 22:34:00.231942 Captured frame from MAC (72:07:34:51:ac:51) to (e2:01:29:ac:0e:87) associated with another peer b6:5d:20:e7:e6:ef(g-k8s-worker-5)
INFO: 2019/02/02 22:34:00.249428 Discovered remote MAC 22:61:62:5f:62:67 at ba:fb:29:f8:cb:84(g-k8s-worker-4)
ERRO: 2019/02/02 22:34:00.249468 Captured frame from MAC (22:61:62:5f:62:67) to (5a:da:3d:bd:85:4b) associated with another peer ba:fb:29:f8:cb:84(g-k8s-worker-4)
ERRO: 2019/02/02 22:34:00.250642 Captured frame from MAC (22:61:62:5f:62:67) to (e2:01:29:ac:0e:87) associated with another peer ba:fb:29:f8:cb:84(g-k8s-worker-4)
INFO: 2019/02/02 22:34:00.642708 Discovered remote MAC 86:d0:59:28:ac:31 at ea:f7:f6:01:b2:4e(g-k8s-master-1)
ERRO: 2019/02/02 22:34:00.642753 Captured frame from MAC (86:d0:59:28:ac:31) to (5a:da:3d:bd:85:4b) associated with another peer ea:f7:f6:01:b2:4e(g-k8s-master-1)
ERRO: 2019/02/02 22:34:00.646578 Captured frame from MAC (86:d0:59:28:ac:31) to (e2:01:29:ac:0e:87) associated with another peer ea:f7:f6:01:b2:4e(g-k8s-master-1)
INFO: 2019/02/02 22:34:01.404363 Discovered remote MAC 0a:38:95:28:d0:99 at de:1e:9e:8d:5d:19(g-k8s-master-0)
ERRO: 2019/02/02 22:34:01.404445 Captured frame from MAC (0a:38:95:28:d0:99) to (72:13:8e:d7:9d:ab) associated with another peer de:1e:9e:8d:5d:19(g-k8s-master-0)
ERRO: 2019/02/02 22:34:01.427405 Captured frame from MAC (ba:37:b3:a0:5e:7f) to (e2:01:29:ac:0e:87) associated with another peer 92:bd:61:89:29:91(g-k8s-worker-6)
ERRO: 2019/02/02 22:34:01.548462 Captured frame from MAC (62:76:77:3d:f2:58) to (da:f5:6f:0f:c2:74) associated with another peer 46:e5:0b:44:b7:67(g-k8s-worker-20)
INFO: 2019/02/02 22:34:01.575870 Discovered remote MAC 5e:55:e0:22:a3:e6 at fe:fc:ec:0d:39:df(g-k8s-worker-1)
INFO: 2019/02/02 22:34:01.642962 Discovered remote MAC d6:5b:35:90:19:eb at de:9d:dc:93:55:1a(g-k8s-worker-18)
ERRO: 2019/02/02 22:34:01.643006 Captured frame from MAC (d6:5b:35:90:19:eb) to (72:13:8e:d7:9d:ab) associated with another peer de:9d:dc:93:55:1a(g-k8s-worker-18)
ERRO: 2019/02/02 22:34:01.915192 Captured frame from MAC (1a:87:ed:8f:de:63) to (da:f5:6f:0f:c2:74) associated with another peer 02:5a:54:30:fa:53(g-k8s-worker-15)
ERRO: 2019/02/02 22:34:02.106808 Captured frame from MAC (5a:19:5f:1e:a6:20) to (da:f5:6f:0f:c2:74) associated with another peer 36:32:51:ad:16:c5(g-k8s-master-2)
INFO: 2019/02/02 22:34:02.356000 Discovered remote MAC 2e:87:81:64:fb:1f at 22:72:12:a3:73:c6(g-k8s-worker-7)
INFO: 2019/02/02 22:34:03.031145 Discovered remote MAC 86:eb:e6:10:98:af at 32:f5:27:3e:b5:e3(g-k8s-worker-19)
ERRO: 2019/02/02 22:34:03.033849 Captured frame from MAC (86:eb:e6:10:98:af) to (e2:01:29:ac:0e:87) associated with another peer 32:f5:27:3e:b5:e3(g-k8s-worker-19)
INFO: 2019/02/02 22:34:03.359627 Discovered remote MAC 42:34:6a:e0:ad:82 at 56:98:7e:b7:2f:ef(g-k8s-worker-12)
ERRO: 2019/02/02 22:34:03.362272 Captured frame from MAC (42:34:6a:e0:ad:82) to (e2:01:29:ac:0e:87) associated with another peer 56:98:7e:b7:2f:ef(g-k8s-worker-12)
ERRO: 2019/02/02 22:34:03.428164 Captured frame from MAC (ba:37:b3:a0:5e:7f) to (da:f5:6f:0f:c2:74) associated with another peer 92:bd:61:89:29:91(g-k8s-worker-6)
ERRO: 2019/02/02 22:34:03.568181 Captured frame from MAC (62:76:77:3d:f2:58) to (26:c3:71:02:3f:e6) associated with another peer 46:e5:0b:44:b7:67(g-k8s-worker-20)
ERRO: 2019/02/02 22:34:03.667752 Captured frame from MAC (86:d0:59:28:ac:31) to (da:f5:6f:0f:c2:74) associated with another peer ea:f7:f6:01:b2:4e(g-k8s-master-1)
ERRO: 2019/02/02 22:34:05.424510 Captured frame from MAC (0a:38:95:28:d0:99) to (da:f5:6f:0f:c2:74) associated with another peer de:1e:9e:8d:5d:19(g-k8s-master-0)
INFO: 2019/02/02 22:34:06.027458 Discovered remote MAC ea:0a:f4:40:d9:70 at 4a:ba:2d:d5:2d:cd(g-k8s-worker-17)
ERRO: 2019/02/02 22:34:06.027508 Captured frame from MAC (ea:0a:f4:40:d9:70) to (da:f5:6f:0f:c2:74) associated with another peer 4a:ba:2d:d5:2d:cd(g-k8s-worker-17)
ERRO: 2019/02/02 22:34:06.030123 Captured frame from MAC (86:eb:e6:10:98:af) to (26:c3:71:02:3f:e6) associated with another peer 32:f5:27:3e:b5:e3(g-k8s-worker-19)
ERRO: 2019/02/02 22:34:06.229839 Captured frame from MAC (72:07:34:51:ac:51) to (da:f5:6f:0f:c2:74) associated with another peer b6:5d:20:e7:e6:ef(g-k8s-worker-5)
ERRO: 2019/02/02 22:34:06.254387 Captured frame from MAC (22:61:62:5f:62:67) to (da:f5:6f:0f:c2:74) associated with another peer ba:fb:29:f8:cb:84(g-k8s-worker-4)
ERRO: 2019/02/02 22:34:06.377492 Captured frame from MAC (42:34:6a:e0:ad:82) to (da:f5:6f:0f:c2:74) associated with another peer 56:98:7e:b7:2f:ef(g-k8s-worker-12)
ERRO: 2019/02/02 22:34:07.918634 Captured frame from MAC (1a:87:ed:8f:de:63) to (e2:01:29:ac:0e:87) associated with another peer 02:5a:54:30:fa:53(g-k8s-worker-15)
ERRO: 2019/02/02 22:34:08.128113 Captured frame from MAC (5a:19:5f:1e:a6:20) to (e2:01:29:ac:0e:87) associated with another peer 36:32:51:ad:16:c5(g-k8s-master-2)
ERRO: 2019/02/02 22:34:09.060750 Captured frame from MAC (86:eb:e6:10:98:af) to (72:13:8e:d7:9d:ab) associated with another peer 32:f5:27:3e:b5:e3(g-k8s-worker-19)
ERRO: 2019/02/02 22:34:09.195100 Captured frame from MAC (ce:81:a6:be:65:b3) to (26:c3:71:02:3f:e6) associated with another peer ca:1b:32:f4:6b:d4(g-k8s-worker-10)
ERRO: 2019/02/02 22:34:09.271036 Captured frame from MAC (22:61:62:5f:62:67) to (26:c3:71:02:3f:e6) associated with another peer ba:fb:29:f8:cb:84(g-k8s-worker-4)
ERRO: 2019/02/02 22:34:10.441241 Captured frame from MAC (ba:37:b3:a0:5e:7f) to (72:13:8e:d7:9d:ab) associated with another peer 92:bd:61:89:29:91(g-k8s-worker-6)
ERRO: 2019/02/02 22:34:12.080662 Captured frame from MAC (86:eb:e6:10:98:af) to (da:f5:6f:0f:c2:74) associated with another peer 32:f5:27:3e:b5:e3(g-k8s-worker-19)
ERRO: 2019/02/02 22:34:14.097625 Captured frame from MAC (ea:0a:f4:40:d9:70) to (72:13:8e:d7:9d:ab) associated with another peer 4a:ba:2d:d5:2d:cd(g-k8s-worker-17)
ERRO: 2019/02/02 22:34:14.145664 Captured frame from MAC (5a:19:5f:1e:a6:20) to (26:c3:71:02:3f:e6) associated with another peer 36:32:51:ad:16:c5(g-k8s-master-2)
ERRO: 2019/02/02 22:34:17.104098 Captured frame from MAC (ea:0a:f4:40:d9:70) to (26:c3:71:02:3f:e6) associated with another peer 4a:ba:2d:d5:2d:cd(g-k8s-worker-17)
ERRO: 2019/02/02 22:34:18.218470 Captured frame from MAC (ce:81:a6:be:65:b3) to (72:13:8e:d7:9d:ab) associated with another peer ca:1b:32:f4:6b:d4(g-k8s-worker-10)
ERRO: 2019/02/02 22:34:20.291382 Captured frame from MAC (22:61:62:5f:62:67) to (72:13:8e:d7:9d:ab) associated with another peer ba:fb:29:f8:cb:84(g-k8s-worker-4)
INFO: 2019/02/02 22:34:31.006613 Discovered remote MAC a6:32:42:ae:f6:af at 6e:a6:6a:81:53:f3(g-k8s-worker-16)
INFO: 2019/02/02 22:35:42.933723 Discovered remote MAC 26:56:6b:4f:fb:f5 at 6e:a6:6a:81:53:f3(g-k8s-worker-16)
ERRO: 2019/02/02 22:35:45.823755 Captured frame from MAC (86:d0:59:28:ac:31) to (72:13:8e:d7:9d:ab) associated with another peer ea:f7:f6:01:b2:4e(g-k8s-master-1)
INFO: 2019/02/02 22:35:48.421882 Discovered remote MAC da:2a:66:72:3d:51 at 6e:a6:6a:81:53:f3(g-k8s-worker-16)
INFO: 2019/02/02 22:36:23.965240 Discovered remote MAC 52:92:9a:93:be:27 at 46:9d:74:35:d5:ec(g-k8s-worker-14)
ERRO: 2019/02/02 22:36:26.459422 Captured frame from MAC (0a:38:95:28:d0:99) to (26:c3:71:02:3f:e6) associated with another peer de:1e:9e:8d:5d:19(g-k8s-master-0)
ERRO: 2019/02/02 22:38:23.607494 Captured frame from MAC (72:07:34:51:ac:51) to (e2:01:29:ac:0e:87) associated with another peer b6:5d:20:e7:e6:ef(g-k8s-worker-5)
ERRO: 2019/02/02 22:38:23.607736 Captured frame from MAC (22:61:62:5f:62:67) to (5a:da:3d:bd:85:4b) associated with another peer ba:fb:29:f8:cb:84(g-k8s-worker-4)
ERRO: 2019/02/02 22:38:23.609401 Captured frame from MAC (86:eb:e6:10:98:af) to (e2:01:29:ac:0e:87) associated with another peer 32:f5:27:3e:b5:e3(g-k8s-worker-19)
ERRO: 2019/02/02 22:38:23.609987 Captured frame from MAC (ce:81:a6:be:65:b3) to (e2:01:29:ac:0e:87) associated with another peer ca:1b:32:f4:6b:d4(g-k8s-worker-10)
ERRO: 2019/02/02 22:38:23.610099 Captured frame from MAC (62:76:77:3d:f2:58) to (e2:01:29:ac:0e:87) associated with another peer 46:e5:0b:44:b7:67(g-k8s-worker-20)
ERRO: 2019/02/02 22:38:23.610388 Captured frame from MAC (42:34:6a:e0:ad:82) to (da:f5:6f:0f:c2:74) associated with another peer 56:98:7e:b7:2f:ef(g-k8s-worker-12)
ERRO: 2019/02/02 22:38:23.611345 Captured frame from MAC (62:76:77:3d:f2:58) to (da:f5:6f:0f:c2:74) associated with another peer 46:e5:0b:44:b7:67(g-k8s-worker-20)
ERRO: 2019/02/02 22:38:23.611480 Captured frame from MAC (ba:37:b3:a0:5e:7f) to (e2:01:29:ac:0e:87) associated with another peer 92:bd:61:89:29:91(g-k8s-worker-6)
ERRO: 2019/02/02 22:38:23.612431 Captured frame from MAC (86:d0:59:28:ac:31) to (e2:01:29:ac:0e:87) associated with another peer ea:f7:f6:01:b2:4e(g-k8s-master-1)
ERRO: 2019/02/02 22:38:23.612926 Captured frame from MAC (22:61:62:5f:62:67) to (72:13:8e:d7:9d:ab) associated with another peer ba:fb:29:f8:cb:84(g-k8s-worker-4)
ERRO: 2019/02/02 22:38:23.613597 Captured frame from MAC (22:61:62:5f:62:67) to (da:f5:6f:0f:c2:74) associated with another peer ba:fb:29:f8:cb:84(g-k8s-worker-4)
ERRO: 2019/02/02 22:38:23.613638 Captured frame from MAC (86:d0:59:28:ac:31) to (5a:da:3d:bd:85:4b) associated with another peer ea:f7:f6:01:b2:4e(g-k8s-master-1)
ERRO: 2019/02/02 22:38:23.613902 Captured frame from MAC (86:eb:e6:10:98:af) to (da:f5:6f:0f:c2:74) associated with another peer 32:f5:27:3e:b5:e3(g-k8s-worker-19)
ERRO: 2019/02/02 22:38:23.614557 Captured frame from MAC (ce:81:a6:be:65:b3) to (72:13:8e:d7:9d:ab) associated with another peer ca:1b:32:f4:6b:d4(g-k8s-worker-10)
ERRO: 2019/02/02 22:38:23.614634 Captured frame from MAC (1a:87:ed:8f:de:63) to (da:f5:6f:0f:c2:74) associated with another peer 02:5a:54:30:fa:53(g-k8s-worker-15)
ERRO: 2019/02/02 22:38:23.614748 Captured frame from MAC (ba:37:b3:a0:5e:7f) to (fa:e7:69:24:51:21) associated with another peer 92:bd:61:89:29:91(g-k8s-worker-6)
ERRO: 2019/02/02 22:38:23.615123 Captured frame from MAC (1a:87:ed:8f:de:63) to (e2:01:29:ac:0e:87) associated with another peer 02:5a:54:30:fa:53(g-k8s-worker-15)
ERRO: 2019/02/02 22:38:23.615222 Captured frame from MAC (86:d0:59:28:ac:31) to (72:13:8e:d7:9d:ab) associated with another peer ea:f7:f6:01:b2:4e(g-k8s-master-1)
ERRO: 2019/02/02 22:38:23.615605 Captured frame from MAC (42:34:6a:e0:ad:82) to (e2:01:29:ac:0e:87) associated with another peer 56:98:7e:b7:2f:ef(g-k8s-worker-12)
ERRO: 2019/02/02 22:38:23.615756 Captured frame from MAC (22:61:62:5f:62:67) to (e2:01:29:ac:0e:87) associated with another peer ba:fb:29:f8:cb:84(g-k8s-worker-4)
INFO: 2019/02/02 22:38:24.725645 Discovered remote MAC 42:32:a9:cc:1a:fd at 6e:a6:6a:81:53:f3(g-k8s-worker-16)
ERRO: 2019/02/02 22:38:26.984186 Captured frame from MAC (1a:87:ed:8f:de:63) to (72:13:8e:d7:9d:ab) associated with another peer 02:5a:54:30:fa:53(g-k8s-worker-15)
ERRO: 2019/02/02 22:38:27.701495 Captured frame from MAC (62:76:77:3d:f2:58) to (72:13:8e:d7:9d:ab) associated with another peer 46:e5:0b:44:b7:67(g-k8s-worker-20)
ERRO: 2019/02/02 22:43:23.604298 Captured frame from MAC (86:d0:59:28:ac:31) to (e2:01:29:ac:0e:87) associated with another peer ea:f7:f6:01:b2:4e(g-k8s-master-1)
ERRO: 2019/02/02 22:43:23.605047 Captured frame from MAC (86:d0:59:28:ac:31) to (5a:da:3d:bd:85:4b) associated with another peer ea:f7:f6:01:b2:4e(g-k8s-master-1)
ERRO: 2019/02/02 22:43:23.605990 Captured frame from MAC (1a:87:ed:8f:de:63) to (da:f5:6f:0f:c2:74) associated with another peer 02:5a:54:30:fa:53(g-k8s-worker-15)
ERRO: 2019/02/02 22:43:23.606527 Captured frame from MAC (1a:87:ed:8f:de:63) to (e2:01:29:ac:0e:87) associated with another peer 02:5a:54:30:fa:53(g-k8s-worker-15)
ERRO: 2019/02/02 22:43:23.606951 Captured frame from MAC (42:34:6a:e0:ad:82) to (e2:01:29:ac:0e:87) associated with another peer 56:98:7e:b7:2f:ef(g-k8s-worker-12)
INFO: 2019/02/02 22:46:34.645276 Discovered remote MAC 3a:33:a1:77:b4:dd at 16:b4:0a:ae:5d:9f(g-k8s-worker-2)
ERRO: 2019/02/02 22:47:56.462545 Captured frame from MAC (22:61:62:5f:62:67) to (72:13:8e:d7:9d:ab) associated with another peer ba:fb:29:f8:cb:84(g-k8s-worker-4)
ERRO: 2019/02/02 22:48:23.610076 Captured frame from MAC (62:76:77:3d:f2:58) to (e2:01:29:ac:0e:87) associated with another peer 46:e5:0b:44:b7:67(g-k8s-worker-20)
ERRO: 2019/02/02 22:48:23.610254 Captured frame from MAC (ba:37:b3:a0:5e:7f) to (e2:01:29:ac:0e:87) associated with another peer 92:bd:61:89:29:91(g-k8s-worker-6)
ERRO: 2019/02/02 22:48:23.610947 Captured frame from MAC (1a:87:ed:8f:de:63) to (da:f5:6f:0f:c2:74) associated with another peer 02:5a:54:30:fa:53(g-k8s-worker-15)
ERRO: 2019/02/02 22:48:23.611544 Captured frame from MAC (72:07:34:51:ac:51) to (e2:01:29:ac:0e:87) associated with another peer b6:5d:20:e7:e6:ef(g-k8s-worker-5)
INFO: 2019/02/02 22:49:02.579482 Discovered remote MAC d6:b4:ac:a8:9c:1c at 46:9d:74:35:d5:ec(g-k8s-worker-14)
INFO: 2019/02/02 22:49:20.943690 Discovered remote MAC d6:5b:35:90:19:eb at de:9d:dc:93:55:1a(g-k8s-worker-18)
INFO: 2019/02/02 22:51:13.988519 Discovered remote MAC 52:a1:95:d3:76:7d at ce:c8:b6:c0:20:70(g-k8s-worker-0)
ERRO: 2019/02/02 22:51:16.587600 Captured frame from MAC (42:34:6a:e0:ad:82) to (e2:01:29:ac:0e:87) associated with another peer 56:98:7e:b7:2f:ef(g-k8s-worker-12)
ERRO: 2019/02/02 22:51:16.664025 Captured frame from MAC (ba:37:b3:a0:5e:7f) to (72:13:8e:d7:9d:ab) associated with another peer 92:bd:61:89:29:91(g-k8s-worker-6)
ERRO: 2019/02/02 22:51:17.216019 Captured frame from MAC (86:d0:59:28:ac:31) to (72:13:8e:d7:9d:ab) associated with another peer ea:f7:f6:01:b2:4e(g-k8s-master-1)
ERRO: 2019/02/02 22:51:18.585282 Captured frame from MAC (42:34:6a:e0:ad:82) to (72:13:8e:d7:9d:ab) associated with another peer 56:98:7e:b7:2f:ef(g-k8s-worker-12)
INFO: 2019/02/02 22:51:19.759898 Discovered remote MAC 22:7f:69:a1:26:ec at 16:b4:0a:ae:5d:9f(g-k8s-worker-2)
INFO: 2019/02/02 22:51:24.235292 Discovered remote MAC 9a:31:bd:61:70:45 at aa:89:98:8c:ac:9f(g-k8s-worker-3)
INFO: 2019/02/02 22:51:27.396062 Discovered remote MAC 5a:19:5f:1e:a6:20 at 36:32:51:ad:16:c5(g-k8s-master-2)
ERRO: 2019/02/02 22:52:25.951007 Captured frame from MAC (1a:87:ed:8f:de:63) to (72:13:8e:d7:9d:ab) associated with another peer 02:5a:54:30:fa:53(g-k8s-worker-15)
ERRO: 2019/02/02 22:52:28.469320 Captured frame from MAC (72:07:34:51:ac:51) to (26:c3:71:02:3f:e6) associated with another peer b6:5d:20:e7:e6:ef(g-k8s-worker-5)
ERRO: 2019/02/02 22:53:23.603614 Captured frame from MAC (72:07:34:51:ac:51) to (26:c3:71:02:3f:e6) associated with another peer b6:5d:20:e7:e6:ef(g-k8s-worker-5)
ERRO: 2019/02/02 22:53:23.604321 Captured frame from MAC (ba:37:b3:a0:5e:7f) to (72:13:8e:d7:9d:ab) associated with another peer 92:bd:61:89:29:91(g-k8s-worker-6)
ERRO: 2019/02/02 22:53:23.604423 Captured frame from MAC (42:34:6a:e0:ad:82) to (e2:01:29:ac:0e:87) associated with another peer 56:98:7e:b7:2f:ef(g-k8s-worker-12)
ERRO: 2019/02/02 22:53:23.605167 Captured frame from MAC (42:34:6a:e0:ad:82) to (72:13:8e:d7:9d:ab) associated with another peer 56:98:7e:b7:2f:ef(g-k8s-worker-12)
ERRO: 2019/02/02 22:53:23.605339 Captured frame from MAC (62:76:77:3d:f2:58) to (e2:01:29:ac:0e:87) associated with another peer 46:e5:0b:44:b7:67(g-k8s-worker-20)
ERRO: 2019/02/02 22:53:23.605595 Captured frame from MAC (ba:37:b3:a0:5e:7f) to (e2:01:29:ac:0e:87) associated with another peer 92:bd:61:89:29:91(g-k8s-worker-6)
ERRO: 2019/02/02 22:53:23.606078 Captured frame from MAC (1a:87:ed:8f:de:63) to (da:f5:6f:0f:c2:74) associated with another peer 02:5a:54:30:fa:53(g-k8s-worker-15)
INFO: 2019/02/02 22:53:24.890813 Discovered remote MAC 52:92:9a:93:be:27 at 46:9d:74:35:d5:ec(g-k8s-worker-14)
ERRO: 2019/02/02 22:53:27.435993 Captured frame from MAC (5a:19:5f:1e:a6:20) to (26:c3:71:02:3f:e6) associated with another peer 36:32:51:ad:16:c5(g-k8s-master-2)
ERRO: 2019/02/02 22:57:26.696924 Captured frame from MAC (62:76:77:3d:f2:58) to (72:13:8e:d7:9d:ab) associated with another peer 46:e5:0b:44:b7:67(g-k8s-worker-20)```

harrycain72 commented 5 years ago

I am also seeing a lot of the same error. But the network seems to work ... It is very confusing ...

I am using Version: 2.5.0 (version check update disabled)

    Service: router
   Protocol: weave 1..2
       Name: 7e:98:57:a0:b6:d5(mykc-dev-blue-k8s-master-nf-3)
 Encryption: enabled

PeerDiscovery: enabled Targets: 4 Connections: 4 (3 established, 1 failed) Peers: 4 (with 12 established connections) TrustedSubnets: none

    Service: ipam
     Status: ready
      Range: 10.233.64.0/18

DefaultSubnet: 10.233.64.0/18

My error log looks similar. The more nodes the more errors ... . Very, very annoying ...

ERRO: 2019/02/02 23:08:37.833752 Captured frame from MAC (3e:89:96:d5:fb:01) to (ba:18:12:53:40:44) associated with another peer 26:5f:ee:b4:31:b6(mykc-dev-blue-k8s-master-nf-2) ERRO: 2019/02/02 23:08:38.464359 Captured frame from MAC (3e:89:96:d5:fb:01) to (7e:9d:8c:fb:20:5c) associated with another peer 26:5f:ee:b4:31:b6(mykc-dev-blue-k8s-master-nf-2) INFO: 2019/02/02 23:08:43.215362 Discovered remote MAC 92:eb:f3:f0:13:6e at 7e:98:57:a0:b6:d5(mykc-dev-blue-k8s-master-nf-3) ERRO: 2019/02/02 23:08:43.215411 Captured frame from MAC (92:eb:f3:f0:13:6e) to (ba:18:12:53:40:44) associated with another peer 7e:98:57:a0:b6:d5(mykc-dev-blue-k8s-master-nf-3) ERRO: 2019/02/02 23:08:43.904890 Captured frame from MAC (92:eb:f3:f0:13:6e) to (7e:9d:8c:fb:20:5c) associated with another peer 7e:98:57:a0:b6:d5(mykc-dev-blue-k8s-master-nf-3)

paleksic commented 5 years ago

Same here I'm getting tons of them but network works, only thing I found is that hairpin is off even though kubelet and weave are configured to enable it.

harrycain72 commented 5 years ago

my yaml setup for the weave pods look like this (created using kubespray)

apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: weave-net labels: name: weave-net namespace: kube-system spec: minReadySeconds: 5 template: metadata: labels: name: weave-net annotations:

Mark pod as critical for rescheduling (Will have no effect starting with kubernetes 1.12)

    scheduler.alpha.kubernetes.io/critical-pod: ''
spec:
  priorityClassName: system-node-critical
  containers:
    - name: weave
      command:
        - /home/weave/launch.sh
      env:
        - name: HOSTNAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: WEAVE_PASSWORD
          valueFrom:
            secretKeyRef:
              name: weave-net
              key: WEAVE_PASSWORD
        - name: CHECKPOINT_DISABLE
          value: "0"
        - name: CONN_LIMIT
          value: "100"
        - name: HAIRPIN_MODE
          value: "true"
        - name: IPALLOC_RANGE
          value: "10.233.64.0/18"
        - name: EXPECT_NPC
          value: "1"
        - name: WEAVE_MTU
          value: "1376"
        - name: NO_MASQ_LOCAL
          value: "1"
      image: docker.io/weaveworks/weave-kube:2.5.0

paleksic commented 5 years ago

We're both using kubespray and weave 2.5.0 configuration is the same. I'm on k8s 1.12 No idea how to troubleshoot this further and why my veth bridge hairpin is off and if that has anything to do with error.

harrycain72 commented 5 years ago

how do you check "veth bridge hairpin is off" ? are you looking into journalctl ?

This is how my journalctl log looks like:

root@mykc-dev-blue-k8s-master-nf-1:~# journalctl | grep hairpin Feb 02 23:39:09 mykc-dev-blue-k8s-master-nf-1 kubelet[14641]: I0202 23:39:09.905985 14641 flags.go:33] FLAG: --hairpin-mode="promiscuous-bridge" Feb 02 23:39:20 mykc-dev-blue-k8s-master-nf-1 kubelet[14746]: I0202 23:39:20.598608 14746 flags.go:33] FLAG: --hairpin-mode="promiscuous-bridge" Feb 02 23:39:31 mykc-dev-blue-k8s-master-nf-1 kubelet[14825]: I0202 23:39:31.068918 14825 flags.go:33] FLAG: --hairpin-mode="promiscuous-bridge" Feb 02 23:39:41 mykc-dev-blue-k8s-master-nf-1 kubelet[14971]: I0202 23:39:41.315733 14971 flags.go:33] FLAG: --hairpin-mode="promiscuous-bridge" Feb 02 23:39:51 mykc-dev-blue-k8s-master-nf-1 kubelet[15074]: I0202 23:39:51.509612 15074 flags.go:33] FLAG: --hairpin-mode="promiscuous-bridge" Feb 02 23:40:01 mykc-dev-blue-k8s-master-nf-1 kubelet[15227]: I0202 23:40:01.229541 15227 flags.go:33] FLAG: --hairpin-mode="promiscuous-bridge" Feb 02 23:40:11 mykc-dev-blue-k8s-master-nf-1 kubelet[15241]: I0202 23:40:11.525293 15241 flags.go:33] FLAG: --hairpin-mode="promiscuous-bridge" Feb 02 23:40:12 mykc-dev-blue-k8s-master-nf-1 kubelet[15241]: W0202 23:40:12.846739 15241 docker_service.go:540] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth" Feb 02 23:40:12 mykc-dev-blue-k8s-master-nf-1 kubelet[15241]: I0202 23:40:12.846765 15241 docker_service.go:236] Hairpin mode set to "hairpin-veth" Feb 02 23:42:10 mykc-dev-blue-k8s-master-nf-1 kubelet[15935]: I0202 23:42:10.739280 15935 flags.go:33] FLAG: --hairpin-mode="promiscuous-bridge" Feb 02 23:42:12 mykc-dev-blue-k8s-master-nf-1 kubelet[15935]: W0202 23:42:12.238495 15935 docker_service.go:540] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth" Feb 02 23:42:12 mykc-dev-blue-k8s-master-nf-1 kubelet[15935]: I0202 23:42:12.238875 15935 docker_service.go:236] Hairpin mode set to "hairpin-veth"

paleksic commented 5 years ago

ip -d link show vethwe-bridge


    link/ether de:7d:50:50:dc:33 brd ff:ff:ff:ff:ff:ff promiscuity 1
    veth
    bridge_slave state forwarding priority 32 cost 2 hairpin off guard off root_block off fastleave off learning on flood on addrgenmode eui64```

bboreham commented 5 years ago

Some discussion at #2877 of how this can happen and be innocuous.

weaveworks / weave

Captured frame from MAC #3591

Mark pod as critical for rescheduling (Will have no effect starting with kubernetes 1.12)