Pods loosing connection to each other on particular node from time to time

JiriHorky commented 5 years ago

Hello,

we are seeing quite strange and (at least for me) hard to debug issue when a Kuberetes node gets into a state where any pod running on it cannot reach any other pods withing the cluster including pods running on the very same node.

We have a node being in this situation right now, so I can provide any debug output if needed (the node is cordened from Kubernetes and all production pods are drained from it).

Let me describe our setup first:

root@ip-10-110-174-111:~# /opt/cni/bin/cni-ipvlan-vpc-k8s-tool eniif 
iface   mac                 id                      subnet                     subnet_cidr       secgrps                  vpc                     ips                                                                                                                                                                                                                         
eth0    0a:2a:d3:92:7e:8c   eni-018ae0468fdf10781   subnet-0e238589d27216701   10.110.174.0/25   [sg-0ccda037d02911c59]   vpc-0f1dab3aaf561ddba   [10.110.174.111]                                                                                                                                                                                                            
eth1    0a:b5:1f:03:d1:d6   eni-0f5769f94c79afb21   subnet-0004ceb2afd7e3ba7   100.96.224.0/19   [sg-0ccda037d02911c59]   vpc-0f1dab3aaf561ddba   [100.96.236.178 100.96.235.253 100.96.255.154 100.96.248.69 100.96.254.43 100.96.253.184 100.96.234.151 100.96.254.206 100.96.244.19 100.96.230.47 100.96.251.97 100.96.239.70 100.96.228.204 100.96.236.91 100.96.225.8]   
eth2    0a:e2:0f:b4:49:4c   eni-0aba7b11eb2077b50   subnet-0004ceb2afd7e3ba7   100.96.224.0/19   [sg-0ccda037d02911c59]   vpc-0f1dab3aaf561ddba   [100.96.254.118 100.96.247.76 100.96.233.85 100.96.228.225 100.96.255.230 100.96.227.61 100.96.230.198 100.96.247.0 100.96.236.129 100.96.255.57 100.96.236.8]

We only have two pods running on the node now:

fluentd-loggly-hb9lf                                             1/1       Running     0          228d      100.96.236.178   ip-10-110-174-111.eu-central-1.compute.internal
jhorky-shell                                                     1/1       Running     0          21m       100.96.254.43    ip-10-110-174-111.eu-central-1.compute.internal

The pods can't see each other even though they are in the same (/19) subnet and running on the same node:

bash-5.0# ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UNKNOWN group default 
    link/ether 0a:b5:1f:03:d1:d6 brd ff:ff:ff:ff:ff:ff
    inet 100.96.254.43/19 brd 100.96.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::8b5:1fff:fe03:d1d6/64 scope link dadfailed tentative 
       valid_lft forever preferred_lft forever
4: veth0@if114: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 0a:70:58:d2:9d:d6 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::870:58ff:fed2:9dd6/64 scope link 
       valid_lft forever preferred_lft forever
bash-5.0# ping 100.96.236.178
PING 100.96.236.178 (100.96.236.178) 56(84) bytes of data.
From 100.96.254.43 icmp_seq=1 Destination Host Unreachable
From 100.96.254.43 icmp_seq=2 Destination Host Unreachable
From 100.96.254.43 icmp_seq=3 Destination Host Unreachable

In a tcpdump (running with -i any) on the compute node, I see these ARP requests but no replies:

15:36:09.107512 ARP, Request who-has 100.96.236.178 tell 100.96.254.43, length 28
15:36:10.129829 ARP, Request who-has 100.96.236.178 tell 100.96.254.43, length 28
15:36:11.153879 ARP, Request who-has 100.96.236.178 tell 100.96.254.43, length 28
15:36:12.177949 ARP, Request who-has 100.96.236.178 tell 100.96.254.43, length 28
15:36:13.201804 ARP, Request who-has 100.96.236.178 tell 100.96.254.43, length 28

When trying to ping the other way around, the situation is very different: The ping works:

PING 100.96.254.43 (100.96.254.43) 56(84) bytes of data.
64 bytes from 100.96.254.43: icmp_seq=1 ttl=64 time=0.168 ms
64 bytes from 100.96.254.43: icmp_seq=2 ttl=64 time=0.051 ms

The tcpdump running on the "debug jhorky" docker shows: just this (no ICMP messages?!?):

15:57:22.065934 ARP, Request who-has 100.96.254.43 tell 100.96.236.178, length 28
15:57:22.066041 ARP, Reply 100.96.254.43 is-at 0a:b5:1f:03:d1:d6, length 28

The tcpdump running on the compute node doesn't show any icmp as well:

15:57:16.241711 ARP, Request who-has 100.96.236.178 tell 10.110.174.111, length 28
15:57:16.241732 ARP, Reply 100.96.236.178 is-at d2:5d:eb:14:bf:f1, length 28

Anyway, right now, I have no idea what more to look at.

Once again, the node is in this state, so I can provide any output needed.

Any help much appreciated.

ngaya-ll commented 5 years ago

@theatrus I think this is the same issue I discussed with you on Gitter a while back where it appears the ENI is not forwarding packets to the ipvlan interface correctly for certain pod IPs. Might be due to some bug or race condition with the ipvlan module for our kernel version. Unloading and reloading the module resolves the issue.

PaulFurtado commented 5 years ago

disclaimer: I don't work on the lyft cni-ipvlan plugin, but I do use it in production on hundreds of nodes so I've debugged a variety of issues over time.

The tcpdump running on the "debug jhorky" docker shows: just this (no ICMP messages?!?):

Note that at the lowest layer, packets are addressed to MAC addresses (be it the mac address of end IP, or the gateway). To discover the MAC address, an ARP broadcast packet is sent out saying "who has this IP?" and an ICMP packet won't be sent until the ARP packet receives a response.

More questions:

Was the pod originally working fine and then suddenly stopped moving packets? Or was it broken from the start?
What does the route table inside both pods show? ip route show table all
- And same on the host?
Arp cache inside the pods? ip neigh show
- And same on the host?
ip addr on the host. (It's possible that something assigned these IPs to the host interfaces and they're stealing the packets t hat would be destined for the pods)
Kernel version? uname -a
dmesg output? Can paste it in a gist if it's long so that the issue isn't cluttered. Also, it's usually not an issue, but run glance through it and make sure there's nothing sensitive in it.

dbyron0 commented 5 years ago

@JiriHorky and I work together, so I'll jump in and add some details.

The jhorky-shell pod is a new one -- one created after we started seeing the problem. Previously existing pods keep working.

inside jorky-shell:

# ip route show table all
default via 10.110.174.111 dev veth0 
10.110.172.0/22 via 100.96.224.1 dev eth0 
10.110.174.111 dev veth0 scope link 
100.96.0.0/16 via 100.96.224.1 dev eth0 
100.96.224.0/19 dev eth0 proto kernel scope link src 100.96.254.43 
100.97.0.0/19 via 100.96.224.1 dev eth0 
broadcast 100.96.224.0 dev eth0 table local proto kernel scope link src 100.96.254.43 
local 100.96.254.43 dev eth0 table local proto kernel scope host src 100.96.254.43 
broadcast 100.96.255.255 dev eth0 table local proto kernel scope link src 100.96.254.43 
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1 
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1 
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1 
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1 
fe80::82a:d3ff:fe92:7e8c dev veth0 metric 1024 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev veth0 proto kernel metric 256 pref medium
unreachable default dev lo proto kernel metric 4294967295 error 4294967195 pref medium
local ::1 dev lo table local proto unspec metric 0 pref medium
local fe80::870:58ff:fed2:9dd6 dev lo table local proto unspec metric 0 pref medium
ff00::/8 dev eth0 table local metric 256 pref medium
ff00::/8 dev veth0 table local metric 256 pref medium
unreachable default dev lo proto kernel metric 4294967295 error 4294967195 pref medium

no ip command inside the fluentd-loggly-hb9lf container, so using nsenter from the host:

$ kubectl get pod fluentd-loggly-hb9lf -o jsonpath='{.status.containerStatuses[0].containerID}' | cut -c 10-21
5c13a573b480

and then on the node:

$ export FLUENTD_PID=$(sudo docker inspect --format '{{ .State.Pid }}' 5c13a573b480)
$ echo $FLUENTD_PID
4163
$ sudo nsenter -t ${FLUENTD_PID} -n ip route show table all
default via 10.110.174.111 dev veth0 
10.110.172.0/22 via 100.96.224.1 dev eth0 
10.110.174.111 dev veth0 scope link 
100.96.0.0/16 via 100.96.224.1 dev eth0 
100.96.224.0/19 dev eth0 proto kernel scope link src 100.96.236.178 
100.97.0.0/19 via 100.96.224.1 dev eth0 
broadcast 100.96.224.0 dev eth0 table local proto kernel scope link src 100.96.236.178 
local 100.96.236.178 dev eth0 table local proto kernel scope host src 100.96.236.178 
broadcast 100.96.255.255 dev eth0 table local proto kernel scope link src 100.96.236.178 
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1 
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1 
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1 
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1 
fe80::82a:d3ff:fe92:7e8c dev veth0 metric 1024  pref medium
fe80::/64 dev eth0 proto kernel metric 256  pref medium
fe80::/64 dev veth0 proto kernel metric 256  pref medium
unreachable default dev lo proto kernel metric 4294967295  error -101 pref medium
local ::1 dev lo table local proto none metric 0  pref medium
local fe80::8b5:1fff:fe03:d1d6 dev lo table local proto none metric 0  pref medium
local fe80::d05d:ebff:fe14:bff1 dev lo table local proto none metric 0  pref medium
ff00::/8 dev eth0 table local metric 256  pref medium
ff00::/8 dev veth0 table local metric 256  pref medium
unreachable default dev lo proto kernel metric 4294967295  error -101 pref medium

and then on the host:

root@ip-10-110-174-111:~#  ip route show table all
10.110.172.0/22 via 100.96.248.69 dev vethd6d10032 table 588 
100.96.0.0/16 via 100.96.248.69 dev vethd6d10032 table 588 
100.97.0.0/19 via 100.96.248.69 dev vethd6d10032 table 588 
10.110.172.0/22 via 100.96.254.43 dev veth433971d5 table 1144 
100.96.0.0/16 via 100.96.254.43 dev veth433971d5 table 1144 
100.97.0.0/19 via 100.96.254.43 dev veth433971d5 table 1144 
10.110.172.0/22 via 100.96.236.178 dev veth3bb00fe4 table 389 
100.96.0.0/16 via 100.96.236.178 dev veth3bb00fe4 table 389 
100.97.0.0/19 via 100.96.236.178 dev veth3bb00fe4 table 389 
default via 10.110.174.1 dev eth0 
10.110.174.0/25 dev eth0 proto kernel scope link src 10.110.174.111 
100.96.236.178 dev veth3bb00fe4 scope link 
100.96.248.69 dev vethd6d10032 scope link 
100.96.254.43 dev veth433971d5 scope link 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
broadcast 10.110.174.0 dev eth0 table local proto kernel scope link src 10.110.174.111 
local 10.110.174.111 dev eth0 table local proto kernel scope host src 10.110.174.111 
broadcast 10.110.174.127 dev eth0 table local proto kernel scope link src 10.110.174.111 
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1 
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1 
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1 
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1 
broadcast 172.17.0.0 dev docker0 table local proto kernel scope link src 172.17.0.1 linkdown 
local 172.17.0.1 dev docker0 table local proto kernel scope host src 172.17.0.1 
broadcast 172.17.255.255 dev docker0 table local proto kernel scope link src 172.17.0.1 linkdown 
fe80::/64 dev eth0 proto kernel metric 256  pref medium
fe80::/64 dev eth1 proto kernel metric 256  pref medium
fe80::/64 dev veth3bb00fe4 proto kernel metric 256  pref medium
fe80::/64 dev eth2 proto kernel metric 256  pref medium
fe80::/64 dev vethd6d10032 proto kernel metric 256  pref medium
fe80::/64 dev docker0 proto kernel metric 256 linkdown  pref medium
fe80::/64 dev veth433971d5 proto kernel metric 256  pref medium
unreachable default dev lo proto kernel metric 4294967295  error -101 pref medium
local ::1 dev lo table local proto none metric 0  pref medium
local fe80::42:4dff:fe6e:8265 dev lo table local proto none metric 0  pref medium
local fe80::82a:d3ff:fe92:7e8c dev lo table local proto none metric 0  pref medium
local fe80::8b5:1fff:fe03:d1d6 dev lo table local proto none metric 0  pref medium
local fe80::8e2:fff:feb4:494c dev lo table local proto none metric 0  pref medium
local fe80::680d:54ff:fe97:81ec dev lo table local proto none metric 0  pref medium
local fe80::8cd2:b6ff:fe7e:fa06 dev lo table local proto none metric 0  pref medium
local fe80::cc29:79ff:fe0e:7424 dev lo table local proto none metric 0  pref medium
ff00::/8 dev eth0 table local metric 256  pref medium
ff00::/8 dev eth1 table local metric 256  pref medium
ff00::/8 dev veth3bb00fe4 table local metric 256  pref medium
ff00::/8 dev eth2 table local metric 256  pref medium
ff00::/8 dev vethd6d10032 table local metric 256  pref medium
ff00::/8 dev docker0 table local metric 256 linkdown  pref medium
ff00::/8 dev veth433971d5 table local metric 256  pref medium
unreachable default dev lo proto kernel metric 4294967295  error -101 pref medium

dbyron0 commented 5 years ago

And now for ip neigh show using nsenter for both pods now:

$ kubectl get pod jhorky-shell -o jsonpath='{.status.containerStatuses[0].containerID}' | cut -c 10-21
a20a2dc6860d

root@ip-10-110-174-111:~# export JHORKY_PID=$(sudo docker inspect --format '{{ .State.Pid }}' a20a2dc6860d)
root@ip-10-110-174-111:~# echo $JHORKY_PID
27750

root@ip-10-110-174-111:~# nsenter -t ${JHORKY_PID} -n ip neigh show
root@ip-10-110-174-111:~# nsenter -t ${FLUENTD_PID} -n ip neigh show
100.96.224.1 dev eth0 lladdr 0a:bd:49:18:1d:5a REACHABLE
10.110.174.111 dev veth0 lladdr 8e:d2:b6:7e:fa:06 STALE
root@ip-10-110-174-111:~# ip neigh show
100.96.248.69 dev vethd6d10032 lladdr 36:d5:bb:8a:66:30 REACHABLE
10.110.174.1 dev eth0 lladdr 0a:9c:72:91:21:24 REACHABLE
10.110.174.39 dev eth0 lladdr 0a:c3:77:78:fc:40 REACHABLE
100.96.236.178 dev veth3bb00fe4 lladdr d2:5d:eb:14:bf:f1 STALE

dbyron0 commented 5 years ago

ip addr on the host:

root@ip-10-110-174-111:~# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 0a:2a:d3:92:7e:8c brd ff:ff:ff:ff:ff:ff
    inet 10.110.174.111/25 brd 10.110.174.127 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::82a:d3ff:fe92:7e8c/64 scope link 
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:4d:6e:82:65 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:4dff:fe6e:8265/64 scope link 
       valid_lft forever preferred_lft forever
4: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 0a:b5:1f:03:d1:d6 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::8b5:1fff:fe03:d1d6/64 scope link 
       valid_lft forever preferred_lft forever
5: veth3bb00fe4@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 8e:d2:b6:7e:fa:06 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::8cd2:b6ff:fe7e:fa06/64 scope link 
       valid_lft forever preferred_lft forever
33: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 0a:e2:0f:b4:49:4c brd ff:ff:ff:ff:ff:ff
    inet6 fe80::8e2:fff:feb4:494c/64 scope link 
       valid_lft forever preferred_lft forever
44: vethd6d10032@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether ce:29:79:0e:74:24 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::cc29:79ff:fe0e:7424/64 scope link 
       valid_lft forever preferred_lft forever
114: veth433971d5@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 6a:0d:54:97:81:ec brd ff:ff:ff:ff:ff:ff link-netnsid 18
    inet6 fe80::680d:54ff:fe97:81ec/64 scope link 
       valid_lft forever preferred_lft forever

dbyron0 commented 5 years ago

root@ip-10-110-174-111:~# uname -a
Linux ip-10-110-174-111 4.9.0-7-amd64 #1 SMP Debian 4.9.110-3+deb9u2 (2018-08-13) x86_64 GNU/Linux

dbyron0 commented 5 years ago

[    0.000000] Linux version 4.9.0-7-amd64 (debian-kernel@lists.debian.org) (gcc version 6.3.0 20170516 (Debian 6.3.0-18+deb9u1) ) #1 SMP Debian 4.9.110-3+deb9u2 (2018-08-13)
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-4.9.0-7-amd64 root=UUID=0cf230ac-6333-41d7-853d-f7ba1683d8b2 ro net.ifnames=0 biosdevname=0 init=/bin/systemd cgroup_enable=memory oops=panic panic=10 console=ttyS0
[    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
[    0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format.
[    0.000000] e820: BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009dfff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009e000-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000efffffff] usable
[    0.000000] BIOS-e820: [mem 0x00000000fc000000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000040fffffff] usable
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 2.7 present.
[    0.000000] DMI: Xen HVM domU, BIOS 4.2.amazon 08/24/2006
[    0.000000] Hypervisor detected: Xen
[    0.000000] Xen version 4.2.
[    0.000000] Xen Platform PCI: I/O protocol version 1
[    0.000000] Netfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated NICs.
[    0.000000] Blkfront and the Xen platform PCI driver have been compiled for this kernel: unplug emulated disks.
               You might have to change the root device
               from /dev/hd[a-d] to /dev/xvd[a-d]
               in your root= kernel command line option
[    0.000000] HVMOP_pagetable_dying not supported
[    0.000000] e820: update [mem 0x00000000-0x00000fff] usable ==> reserved
[    0.000000] e820: remove [mem 0x000a0000-0x000fffff] usable
[    0.000000] e820: last_pfn = 0x410000 max_arch_pfn = 0x400000000
[    0.000000] MTRR default type: write-back
[    0.000000] MTRR fixed ranges enabled:
[    0.000000]   00000-9FFFF write-back
[    0.000000]   A0000-BFFFF write-combining
[    0.000000]   C0000-FFFFF write-back
[    0.000000] MTRR variable ranges enabled:
[    0.000000]   0 base 0000F0000000 mask 3FFFF8000000 uncachable
[    0.000000]   1 base 0000F8000000 mask 3FFFFC000000 uncachable
[    0.000000]   2 disabled
[    0.000000]   3 disabled
[    0.000000]   4 disabled
[    0.000000]   5 disabled
[    0.000000]   6 disabled
[    0.000000]   7 disabled
[    0.000000] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WC  UC- WT  
[    0.000000] e820: last_pfn = 0xf0000 max_arch_pfn = 0x400000000
[    0.000000] found SMP MP-table at [mem 0x000fbc50-0x000fbc5f] mapped at [ffff8d4f400fbc50]
[    0.000000] Base memory trampoline at [ffff8d4f40098000] 98000 size 24576
[    0.000000] Using GB pages for direct mapping
[    0.000000] BRK [0x27773b000, 0x27773bfff] PGTABLE
[    0.000000] BRK [0x27773c000, 0x27773cfff] PGTABLE
[    0.000000] BRK [0x27773d000, 0x27773dfff] PGTABLE
[    0.000000] BRK [0x27773e000, 0x27773efff] PGTABLE
[    0.000000] BRK [0x27773f000, 0x27773ffff] PGTABLE
[    0.000000] BRK [0x277740000, 0x277740fff] PGTABLE
[    0.000000] RAMDISK: [mem 0x35aa7000-0x36d4afff]
[    0.000000] ACPI: Early table checksum verification disabled
[    0.000000] ACPI: RSDP 0x00000000000EA020 000024 (v02 Xen   )
[    0.000000] ACPI: XSDT 0x00000000FC00E2A0 000054 (v01 Xen    HVM      00000000 HVML 00000000)
[    0.000000] ACPI: FACP 0x00000000FC00DF60 0000F4 (v04 Xen    HVM      00000000 HVML 00000000)
[    0.000000] ACPI: DSDT 0x00000000FC0021C0 00BD19 (v02 Xen    HVM      00000000 INTL 20090123)
[    0.000000] ACPI: FACS 0x00000000FC002180 000040
[    0.000000] ACPI: FACS 0x00000000FC002180 000040
[    0.000000] ACPI: APIC 0x00000000FC00E060 0000D8 (v02 Xen    HVM      00000000 HVML 00000000)
[    0.000000] ACPI: HPET 0x00000000FC00E1B0 000038 (v01 Xen    HVM      00000000 HVML 00000000)
[    0.000000] ACPI: WAET 0x00000000FC00E1F0 000028 (v01 Xen    HVM      00000000 HVML 00000000)
[    0.000000] ACPI: SSDT 0x00000000FC00E220 000031 (v02 Xen    HVM      00000000 INTL 20090123)
[    0.000000] ACPI: SSDT 0x00000000FC00E260 000031 (v02 Xen    HVM      00000000 INTL 20090123)
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] No NUMA configuration found
[    0.000000] Faking a node at [mem 0x0000000000000000-0x000000040fffffff]
[    0.000000] NODE_DATA(0) allocated [mem 0x40fffb000-0x40fffffff]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
[    0.000000]   DMA32    [mem 0x0000000001000000-0x00000000ffffffff]
[    0.000000]   Normal   [mem 0x0000000100000000-0x000000040fffffff]
[    0.000000]   Device   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000000001000-0x000000000009dfff]
[    0.000000]   node   0: [mem 0x0000000000100000-0x00000000efffffff]
[    0.000000]   node   0: [mem 0x0000000100000000-0x000000040fffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x000000040fffffff]
[    0.000000] On node 0 totalpages: 4194205
[    0.000000]   DMA zone: 64 pages used for memmap
[    0.000000]   DMA zone: 21 pages reserved
[    0.000000]   DMA zone: 3997 pages, LIFO batch:0
[    0.000000]   DMA32 zone: 15296 pages used for memmap
[    0.000000]   DMA32 zone: 978944 pages, LIFO batch:31
[    0.000000]   Normal zone: 50176 pages used for memmap
[    0.000000]   Normal zone: 3211264 pages, LIFO batch:31
[    0.000000] ACPI: PM-Timer IO Port: 0xb008
[    0.000000] ACPI: Local APIC address 0xfee00000
[    0.000000] IOAPIC[0]: apic_id 1, version 17, address 0xfec00000, GSI 0-47
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 low level)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 low level)
[    0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 low level)
[    0.000000] ACPI: IRQ0 used by override.
[    0.000000] ACPI: IRQ5 used by override.
[    0.000000] ACPI: IRQ9 used by override.
[    0.000000] ACPI: IRQ10 used by override.
[    0.000000] ACPI: IRQ11 used by override.
[    0.000000] Using ACPI (MADT) for SMP configuration information
[    0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000
[    0.000000] smpboot: Allowing 15 CPUs, 11 hotplug CPUs
[    0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff]
[    0.000000] PM: Registered nosave memory: [mem 0x0009e000-0x0009ffff]
[    0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000dffff]
[    0.000000] PM: Registered nosave memory: [mem 0x000e0000-0x000fffff]
[    0.000000] PM: Registered nosave memory: [mem 0xf0000000-0xfbffffff]
[    0.000000] PM: Registered nosave memory: [mem 0xfc000000-0xffffffff]
[    0.000000] e820: [mem 0xf0000000-0xfbffffff] available for PCI devices
[    0.000000] Booting paravirtualized kernel on Xen HVM
[    0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645519600211568 ns
[    0.000000] setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:15 nr_node_ids:1
[    0.000000] percpu: Embedded 35 pages/cpu @ffff8d533f200000 s105304 r8192 d29864 u262144
[    0.000000] pcpu-alloc: s105304 r8192 d29864 u262144 alloc=1*2097152
[    0.000000] pcpu-alloc: [0] 00 01 02 03 04 05 06 07 [0] 08 09 10 11 12 13 14 -- 
[    0.000000] xen: PV spinlocks enabled
[    0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes)
[    0.000000] Built 1 zonelists in Node order, mobility grouping on.  Total pages: 4128648
[    0.000000] Policy zone: Normal
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-4.9.0-7-amd64 root=UUID=0cf230ac-6333-41d7-853d-f7ba1683d8b2 ro net.ifnames=0 biosdevname=0 init=/bin/systemd cgroup_enable=memory oops=panic panic=10 console=ttyS0
[    0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes)
[    0.000000] Calgary: detecting Calgary via BIOS EBDA area
[    0.000000] Calgary: Unable to locate Rio Grande table in EBDA - bailing!
[    0.000000] Memory: 16411720K/16776820K available (6250K kernel code, 1159K rwdata, 2868K rodata, 1420K init, 688K bss, 365100K reserved, 0K cma-reserved)
[    0.000000] Kernel/User page tables isolation: enabled
[    0.000000] Hierarchical RCU implementation.
[    0.000000]  Build-time adjustment of leaf fanout to 64.
[    0.000000]  RCU restricting CPUs from NR_CPUS=512 to nr_cpu_ids=15.
[    0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=64, nr_cpu_ids=15
[    0.000000] NR_IRQS:33024 nr_irqs:952 16
[    0.000000] xen:events: Using 2-level ABI
[    0.000000] xen:events: Xen HVM callback vector for event delivery is enabled
[    0.000000] Console: colour VGA+ 80x25
[    0.000000] Cannot get hvm parameter CONSOLE_EVTCHN (18): -22!
[    0.000000] console [ttyS0] enabled
[    0.000000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 30580167144 ns
[    0.000000] hpet clockevent registered
[    0.000000] tsc: Fast TSC calibration using PIT
[    0.000000] tsc: Detected 2300.034 MHz processor
[    0.028000] Calibrating delay loop (skipped), value calculated using timer frequency.. 4600.09 BogoMIPS (lpj=9200192)
[    0.032003] pid_max: default: 32768 minimum: 301
[    0.036038] ACPI: Core revision 20160831
[    0.047651] ACPI: 3 ACPI AML tables successfully acquired and loaded
[    0.052076] Security Framework initialized
[    0.056003] Yama: disabled by default; enable with sysctl kernel.yama.*
[    0.064015] AppArmor: AppArmor disabled by boot time parameter
[    0.068743] Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes)
[    0.080042] Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes)
[    0.085187] Mount-cache hash table entries: 32768 (order: 6, 262144 bytes)
[    0.092019] Mountpoint-cache hash table entries: 32768 (order: 6, 262144 bytes)
[    0.100354] CPU: Physical Processor ID: 0
[    0.104003] CPU: Processor Core ID: 0
[    0.108020] mce: CPU supports 2 MCE banks
[    0.111796] Last level iTLB entries: 4KB 64, 2MB 8, 4MB 8
[    0.116003] Last level dTLB entries: 4KB 64, 2MB 0, 4MB 0, 1GB 4
[    0.120005] Spectre V2 : Mitigation: Full generic retpoline
[    0.124004] Speculative Store Bypass: Vulnerable
[    0.136999] Freeing SMP alternatives memory: 24K
[    0.145084] ftrace: allocating 25269 entries in 99 pages
[    0.180106] smpboot: Max logical packages: 8
[    0.184832] x2apic: IRQ remapping doesn't support X2APIC mode
[    0.192003] Switched APIC routing to physical flat.
[    0.198734] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=0 pin2=0
[    0.246909] clocksource: xen: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
[    0.256010] Xen: using vcpuop timer interface
[    0.256018] installing Xen timer for CPU 0
[    0.260102] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz (family: 0x6, model: 0x4f, stepping: 0x1)
[    0.267937] cpu 0 spinlock event irq 53
[    0.268007] Performance Events: unsupported p6 CPU model 79 no PMU driver, software events only.
[    0.275916] NMI watchdog: disabled (cpu0): hardware events not enabled
[    0.276008] NMI watchdog: Shutting down hard lockup detector on all cpus
[    0.280154] installing Xen timer for CPU 1
[    0.284070] x86: Booting SMP configuration:
[    0.287570] .... node  #0, CPUs:        #1
[    0.291547] cpu 1 spinlock event irq 59
[    0.292129] installing Xen timer for CPU 2
[    0.296065]   #2
[    0.300036] cpu 2 spinlock event irq 65
[    0.302695] installing Xen timer for CPU 3
[    0.304063]   #3
[    0.308055] cpu 3 spinlock event irq 71
[    0.310705] x86: Booted up 1 node, 4 CPUs
[    0.312006] smpboot: Total of 4 processors activated (18400.38 BogoMIPS)
[    0.316912] devtmpfs: initialized
[    0.320075] x86/mm: Memory block size: 128MB
[    0.326931] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 7645041785100000 ns
[    0.328026] futex hash table entries: 4096 (order: 6, 262144 bytes)
[    0.332147] pinctrl core: initialized pinctrl subsystem
[    0.336802] NET: Registered protocol family 16
[    0.348007] cpuidle: using governor ladder
[    0.364007] cpuidle: using governor menu
[    0.367898] ACPI: bus type PCI registered
[    0.372011] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[    0.376640] PCI: Using configuration type 1 for base access
[    0.392076] HugeTLB registered 1 GB page size, pre-allocated 0 pages
[    0.400011] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    0.404126] random: fast init done
[    0.404148] ACPI: Added _OSI(Module Device)
[    0.404150] ACPI: Added _OSI(Processor Device)
[    0.404151] ACPI: Added _OSI(3.0 _SCP Extensions)
[    0.404151] ACPI: Added _OSI(Processor Aggregator Device)
[    0.404465] xen: --> pirq=16 -> irq=9 (gsi=9)
[    0.429828] ACPI: Interpreter enabled
[    0.432025] ACPI: (supports S0 S3 S4 S5)
[    0.435943] ACPI: Using IOAPIC for interrupt routing
[    0.440048] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[    0.522598] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[    0.528022] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI]
[    0.532022] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM
[    0.540021] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
[    0.549380] acpiphp: Slot [0] registered
[    0.557904] acpiphp: Slot [3] registered
[    0.560695] acpiphp: Slot [4] registered
[    0.564688] acpiphp: Slot [5] registered
[    0.568697] acpiphp: Slot [6] registered
[    0.572678] acpiphp: Slot [7] registered
[    0.576270] acpiphp: Slot [8] registered
[    0.580727] acpiphp: Slot [9] registered
[    0.588632] acpiphp: Slot [10] registered
[    0.592672] acpiphp: Slot [11] registered
[    0.596686] acpiphp: Slot [12] registered
[    0.600664] acpiphp: Slot [13] registered
[    0.604670] acpiphp: Slot [14] registered
[    0.608307] acpiphp: Slot [15] registered
[    0.612801] acpiphp: Slot [16] registered
[    0.616681] acpiphp: Slot [17] registered
[    0.620674] acpiphp: Slot [18] registered
[    0.624675] acpiphp: Slot [19] registered
[    0.632686] acpiphp: Slot [20] registered
[    0.636732] acpiphp: Slot [21] registered
[    0.640511] acpiphp: Slot [22] registered
[    0.643881] acpiphp: Slot [23] registered
[    0.648649] acpiphp: Slot [24] registered
[    0.652706] acpiphp: Slot [25] registered
[    0.656710] acpiphp: Slot [26] registered
[    0.660750] acpiphp: Slot [27] registered
[    0.664726] acpiphp: Slot [28] registered
[    0.668669] acpiphp: Slot [29] registered
[    0.672674] acpiphp: Slot [30] registered
[    0.680680] acpiphp: Slot [31] registered
[    0.684563] PCI host bridge to bus 0000:00
[    0.688008] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
[    0.692007] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
[    0.700008] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
[    0.708008] pci_bus 0000:00: root bus resource [mem 0xf0000000-0xfbffffff window]
[    0.712008] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.720318] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000
[    0.723507] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100
[    0.727814] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180
[    0.730147] pci 0000:00:01.1: reg 0x20: [io  0xc100-0xc10f]
[    0.731183] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io  0x01f0-0x01f7]
[    0.736006] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io  0x03f6]
[    0.744005] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io  0x0170-0x0177]
[    0.748005] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io  0x0376]
[    0.757235] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000
[    0.757321] * Found PM-Timer Bug on the chipset. Due to workarounds for a bug,
               * this clock source is slow. Consider trying other clock sources
[    0.770555] pci 0000:00:01.3: quirk: [io  0xb000-0xb03f] claimed by PIIX4 ACPI
[    0.777667] pci 0000:00:02.0: [1013:00b8] type 00 class 0x030000
[    0.778220] pci 0000:00:02.0: reg 0x10: [mem 0xf0000000-0xf1ffffff pref]
[    0.778644] pci 0000:00:02.0: reg 0x14: [mem 0xf3008000-0xf3008fff]
[    0.782116] pci 0000:00:03.0: [8086:10ed] type 00 class 0x020000
[    0.783137] pci 0000:00:03.0: reg 0x10: [mem 0xf3000000-0xf3003fff 64bit pref]
[    0.784278] pci 0000:00:03.0: reg 0x1c: [mem 0xf3004000-0xf3007fff 64bit pref]
[    0.789644] pci 0000:00:1f.0: [5853:0001] type 00 class 0xff8000
[    0.790371] pci 0000:00:1f.0: reg 0x10: [io  0xc000-0xc0ff]
[    0.790864] pci 0000:00:1f.0: reg 0x14: [mem 0xf2000000-0xf2ffffff pref]
[    0.794744] ACPI: PCI Interrupt Link [LNKA] (IRQs *5 10 11)
[    0.802473] ACPI: PCI Interrupt Link [LNKB] (IRQs 5 *10 11)
[    0.810490] ACPI: PCI Interrupt Link [LNKC] (IRQs 5 10 *11)
[    0.818464] ACPI: PCI Interrupt Link [LNKD] (IRQs *5 10 11)
[    0.854793] ACPI: Enabled 2 GPEs in block 00 to 0F
[    0.860129] xen:balloon: Initialising balloon driver
[    0.864034] xen_balloon: Initialising balloon driver
[    0.868091] vgaarb: setting as boot device: PCI:0000:00:02.0
[    0.872000] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
[    0.884010] vgaarb: loaded
[    0.886590] vgaarb: bridge control possible 0000:00:02.0
[    0.888098] PCI: Using ACPI for IRQ routing
[    0.891821] PCI: pci_cache_line_size set to 64 bytes
[    0.892809] e820: reserve RAM buffer [mem 0x0009e000-0x0009ffff]
[    0.892971] HPET: 3 timers in total, 0 timers will be used for per-cpu timer
[    0.900024] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0
[    0.904692] hpet0: 3 comparators, 64-bit 62.500000 MHz counter
[    0.912039] clocksource: Switched to clocksource xen
[    0.923329] VFS: Disk quotas dquot_6.6.0
[    0.927046] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.933461] pnp: PnP ACPI init
[    0.936413] system 00:00: [mem 0x00000000-0x0009ffff] could not be reserved
[    0.942831] system 00:00: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.942934] system 00:01: [io  0x08a0-0x08a3] has been reserved
[    0.948391] system 00:01: [io  0x0cc0-0x0ccf] has been reserved
[    0.953824] system 00:01: [io  0x04d0-0x04d1] has been reserved
[    0.959226] system 00:01: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.959256] xen: --> pirq=17 -> irq=8 (gsi=8)
[    0.959279] pnp 00:02: Plug and Play ACPI device, IDs PNP0b00 (active)
[    0.959300] xen: --> pirq=18 -> irq=12 (gsi=12)
[    0.959317] pnp 00:03: Plug and Play ACPI device, IDs PNP0f13 (active)
[    0.959334] xen: --> pirq=19 -> irq=1 (gsi=1)
[    0.959350] pnp 00:04: Plug and Play ACPI device, IDs PNP0303 PNP030b (active)
[    0.959368] xen: --> pirq=20 -> irq=6 (gsi=6)
[    0.959370] pnp 00:05: [dma 2]
[    0.959386] pnp 00:05: Plug and Play ACPI device, IDs PNP0700 (active)
[    0.959409] xen: --> pirq=21 -> irq=4 (gsi=4)
[    0.959424] pnp 00:06: Plug and Play ACPI device, IDs PNP0501 (active)
[    0.959469] system 00:07: [io  0x10c0-0x1141] has been reserved
[    0.963920] system 00:07: [io  0xb044-0xb047] has been reserved
[    0.967369] system 00:07: Plug and Play ACPI device, IDs PNP0c02 (active)
[    0.990913] pnp: PnP ACPI: found 8 devices
[    1.001529] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[    1.007939] pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7 window]
[    1.007941] pci_bus 0000:00: resource 5 [io  0x0d00-0xffff window]
[    1.007943] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]
[    1.007944] pci_bus 0000:00: resource 7 [mem 0xf0000000-0xfbffffff window]
[    1.008269] NET: Registered protocol family 2
[    1.010221] TCP established hash table entries: 131072 (order: 8, 1048576 bytes)
[    1.016093] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    1.021320] TCP: Hash tables configured (established 131072 bind 65536)
[    1.026343] UDP hash table entries: 8192 (order: 6, 262144 bytes)
[    1.031025] UDP-Lite hash table entries: 8192 (order: 6, 262144 bytes)
[    1.033865] NET: Registered protocol family 1
[    1.036973] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
[    1.041172] pci 0000:00:01.0: PIIX3: Enabling Passive Release
[    1.045343] pci 0000:00:01.0: Activating ISA DMA hang workarounds
[    1.050457] pci 0000:00:02.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
[    1.053996] PCI: CLS 0 bytes, default 64
[    1.054059] Unpacking initramfs...
[    1.352548] Freeing initrd memory: 19088K
[    1.356004] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    1.361286] software IO TLB [mem 0xec000000-0xf0000000] (64MB) mapped at [ffff8d502c000000-ffff8d502fffffff]
[    1.369927] audit: initializing netlink subsys (disabled)
[    1.374468] audit: type=2000 audit(1539342357.010:1): initialized
[    1.380006] workingset: timestamp_bits=40 max_order=22 bucket_order=0
[    1.385576] zbud: loaded
[    1.389138] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 250)
[    1.395401] io scheduler noop registered
[    1.398796] io scheduler deadline registered
[    1.402408] io scheduler cfq registered (default)
[    1.406556] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    1.411380] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
[    1.417098] intel_idle: does not run on family 6 model 79
[    1.417303] GHES: HEST is not enabled!
[    1.420991] xen: --> pirq=22 -> irq=47 (gsi=47)
[    1.421139] xen:grant_table: Grant tables using version 1 layout
[    1.426420] Grant table initialized
[    1.429436] Cannot get hvm parameter CONSOLE_EVTCHN (18): -22!
[    1.434064] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    1.469536] 00:06: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[    1.476601] Linux agpgart interface v0.103
[    1.480122] AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>
[    1.485059] AMD IOMMUv2 functionality not available on this system
[    1.490159] i8042: PNP: PS/2 Controller [PNP0303:PS2K,PNP0f13:PS2M] at 0x60,0x64 irq 1,12
[    1.499549] serio: i8042 KBD port at 0x60,0x64 irq 1
[    1.503401] serio: i8042 AUX port at 0x60,0x64 irq 12
[    1.507605] mousedev: PS/2 mouse device common for all mice
[    1.512310] rtc_cmos 00:02: rtc core: registered rtc_cmos as rtc0
[    1.512986] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input0
[    1.523694] rtc_cmos 00:02: alarms up to one day, 114 bytes nvram, hpet irqs
[    1.529282] ledtrig-cpu: registered to indicate activity on CPUs
[    1.534206] NET: Registered protocol family 10
[    1.538029] mip6: Mobile IPv6
[    1.540381] NET: Registered protocol family 17
[    1.543788] mpls_gso: MPLS GSO support
[    1.547377] registered taskstats version 1
[    1.550839] zswap: loaded using pool lzo/zbud
[    1.554783] ima: No TPM chip found, activating TPM-bypass!
[    1.559109] ima: Allocated hash algorithm: sha256
[    1.563206] xenbus_probe_frontend: Device with no driver: device/vbd/51712
[    1.568917] xenbus_probe_frontend: Device with no driver: device/pci/0
[    1.574346] rtc_cmos 00:02: setting system clock to 2018-10-12 11:05:57 UTC (1539342357)
[    1.580973] PM: Hibernation image not present or could not be loaded.
[    1.790089] Freeing unused kernel memory: 1420K
[    1.793727] Write protecting the kernel read-only data: 12288k
[    1.799469] Freeing unused kernel memory: 1928K
[    1.805220] Freeing unused kernel memory: 1228K
[    1.812643] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[    1.826714] random: systemd-udevd: uninitialized urandom read (16 bytes read)
[    1.827529] random: udevadm: uninitialized urandom read (16 bytes read)
[    1.827558] random: udevadm: uninitialized urandom read (16 bytes read)
[    1.871092] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 3.2.2-k
[    1.878529] ixgbevf: Copyright (c) 2009 - 2015 Intel Corporation.
[    1.884665] SCSI subsystem initialized
[    1.889988] AVX2 version of gcm_enc/dec engaged.
[    1.893665] AES CTR mode by8 optimization enabled
[    1.898858] ixgbevf 0000:00:03.0: 0a:2a:d3:92:7e:8c
[    1.902898] ixgbevf 0000:00:03.0: MAC: 1
[    1.906169] ixgbevf 0000:00:03.0: Intel(R) 82599 Virtual Function
[    1.911497] libata version 3.00 loaded.
[    1.912022] ata_piix 0000:00:01.1: version 2.13
[    1.913375] scsi host0: ata_piix
[    1.920556] scsi host1: ata_piix
[    1.923599] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc100 irq 14
[    1.923776] blkfront: xvda: flush diskcache: enabled; persistent grants: disabled; indirect descriptors: enabled;
[    1.931894] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc108 irq 15
[    2.021870] GPT:Primary header thinks Alt. header is not at the end of the disk.
[    2.027458] GPT:16777215 != 268435455
[    2.030257] GPT:Alternate GPT header not at the end of the disk.
[    2.034687] GPT:16777215 != 268435455
[    2.037458] GPT: Use GNU Parted to correct GPT errors.
[    2.041215]  xvda: xvda1 xvda2
[    2.396124] tsc: Refined TSC clocksource calibration: 2300.000 MHz
[    2.400675] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x212735223b2, max_idle_ns: 440795277976 ns
[    3.004074] raid6: sse2x1   gen()  8783 MB/s
[    3.072076] raid6: sse2x1   xor()  6571 MB/s
[    3.140074] raid6: sse2x2   gen() 10717 MB/s
[    3.208068] raid6: sse2x2   xor()  7157 MB/s
[    3.276073] raid6: sse2x4   gen() 12556 MB/s
[    3.344069] raid6: sse2x4   xor()  8794 MB/s
[    3.412073] raid6: avx2x1   gen() 17017 MB/s
[    3.480075] raid6: avx2x2   gen() 19663 MB/s
[    3.548071] raid6: avx2x4   gen() 22598 MB/s
[    3.551222] raid6: using algorithm avx2x4 gen() 22598 MB/s
[    3.555115] raid6: using avx2x2 recovery algorithm
[    3.557664] xor: automatically using best checksumming function   avx       
[    3.568561] Btrfs loaded, crc32c=crc32c-intel
[    3.781316] EXT4-fs (xvda2): mounted filesystem with ordered data mode. Opts: (null)
[   10.379769] ip_tables: (C) 2000-2006 Netfilter Core Team
[   10.589581] systemd[1]: systemd 232 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN)
[   10.599662] systemd[1]: Detected virtualization xen.
[   10.603828] systemd[1]: Detected architecture x86-64.
[   10.781855] systemd[1]: Set hostname to <ip-172-31-24-51>.
[   14.073457] systemd[1]: Listening on Journal Socket (/dev/log).
[   14.080179] systemd[1]: Listening on udev Kernel Socket.
[   14.085988] systemd[1]: Listening on RPCbind Server Activation Socket.
[   14.093035] systemd[1]: Listening on LVM2 metadata daemon socket.
[   14.099497] systemd[1]: Listening on udev Control Socket.
[   14.106118] systemd[1]: Reached target Swap.
[   14.536461] EXT4-fs (xvda2): re-mounted. Opts: (null)
[   14.681045] systemd-journald[214]: Received request to flush runtime journal from PID 1
[   14.989695] RPC: Registered named UNIX socket transport module.
[   14.992931] RPC: Registered udp transport module.
[   14.995403] RPC: Registered tcp transport module.
[   14.997970] RPC: Registered tcp NFSv4.1 backchannel transport module.
[   15.730420] random: crng init done
[   15.733782] random: 7 urandom warning(s) missed due to ratelimiting
[   16.260710] RAPL PMU: API unit is 2^-32 Joules, 3 fixed counters, 655360 ms ovfl timer
[   16.266246] RAPL PMU: hw unit of domain pp0-core 2^-14 Joules
[   16.270237] RAPL PMU: hw unit of domain package 2^-14 Joules
[   16.274218] RAPL PMU: hw unit of domain dram 2^-16 Joules
[   16.449380] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input2
[   16.454886] ACPI: Power Button [PWRF]
[   16.457807] input: Sleep Button as /devices/LNXSYSTM:00/LNXSLPBN:00/input/input3
[   16.462670] ACPI: Sleep Button [SLPF]
[   16.748929] EDAC MC: Ver: 3.0.0
[   16.753052] [drm] Initialized
[   16.868210] EDAC sbridge: Seeking for: PCI ID 8086:6fa0
[   16.868214] EDAC sbridge:  Ver: 1.1.1 
[   16.873523] intel_rapl: Found RAPL domain package
[   16.877706] intel_rapl: Found RAPL domain dram
[   16.881513] intel_rapl: DRAM domain energy unit 15300pj
[   17.126001] [TTM] Zone  kernel: Available graphics memory: 8217704 kiB
[   17.129636] [TTM] Zone   dma32: Available graphics memory: 2097152 kiB
[   17.133330] [TTM] Initializing pool allocator
[   17.136205] [TTM] Initializing DMA pool allocator
[   17.139557] [drm] fb mappable at 0xF0000000
[   17.142053] [drm] vram aper at 0xF0000000
[   17.144460] [drm] size 33554432
[   17.146467] [drm] fb depth is 24
[   17.148507] [drm]    pitch is 3072
[   17.150766] fbcon: cirrusdrmfb (fb0) is primary device
[   17.214713] Console: switching to colour frame buffer device 128x48
[   17.271790] cirrus 0000:00:02.0: fb0: cirrusdrmfb frame buffer device
[   17.292109] [drm] Initialized cirrus 1.0.0 20110418 for 0000:00:02.0 on minor 0
[   18.630915] ixgbevf 0000:00:03.0: NIC Link is Up 10 Gbps
[   18.632441] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[   18.637000] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[   28.787614] EXT4-fs (xvda2): resizing filesystem from 2096891 to 33554171 blocks
[   29.692645] EXT4-fs (xvda2): resized filesystem to 33554171
[   42.338758] bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
[   42.459521] Bridge firewalling registered
[   42.534672] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
[   43.338635] Initializing XFRM netlink socket
[   43.522863] IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready
[   49.445920] systemd[1]: apt-daily.timer: Adding 10h 3min 28.584155s random time.
[   88.381559] systemd[1]: apt-daily.timer: Adding 6h 34min 24.411802s random time.
[   88.658358] systemd[1]: apt-daily.timer: Adding 4min 1.882642s random time.
[   88.716898] systemd[1]: apt-daily.timer: Adding 1h 31.635791s random time.
[   88.771574] systemd[1]: apt-daily.timer: Adding 5h 11min 38.815729s random time.
[   88.856565] systemd[1]: apt-daily.timer: Adding 1h 51min 2.532934s random time.
[   89.871048] systemd[1]: apt-daily.timer: Adding 34min 4.929689s random time.
[   90.143584] systemd[1]: apt-daily.timer: Adding 10h 19min 12.989427s random time.
[   90.231624] systemd[1]: apt-daily.timer: Adding 1h 40min 968.053ms random time.
[   91.187772] systemd[1]: apt-daily.timer: Adding 8h 46min 24.755892s random time.
[   91.749226] systemd[1]: apt-daily.timer: Adding 4h 13min 10.439023s random time.
[   93.407535] systemd[1]: apt-daily.timer: Adding 11h 18min 30.806316s random time.
[   93.472798] systemd[1]: apt-daily.timer: Adding 10h 25.081121s random time.
[   93.537399] systemd[1]: apt-daily.timer: Adding 9h 1min 57.345863s random time.
[   93.597975] systemd[1]: apt-daily.timer: Adding 29min 1.413092s random time.
[   93.802323] systemd[1]: apt-daily.timer: Adding 2h 34min 15.126427s random time.
[   93.882321] systemd[1]: apt-daily.timer: Adding 5min 47.144705s random time.
[   93.965615] systemd[1]: apt-daily.timer: Adding 1h 22min 37.530006s random time.
[   94.035538] systemd[1]: apt-daily.timer: Adding 3h 56min 12.818969s random time.
[   94.099536] systemd[1]: apt-daily.timer: Adding 11h 28min 56.013851s random time.
[   94.209393] systemd[1]: apt-daily.timer: Adding 8h 58min 20.625683s random time.
[  102.161451] Netfilter messages via NETLINK v0.30.
[  108.110072] pci 0000:00:04.0: [8086:10ed] type 00 class 0x020000
[  108.111075] pci 0000:00:04.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit pref]
[  108.112034] pci 0000:00:04.0: reg 0x1c: [mem 0x00000000-0x00003fff 64bit pref]
[  108.115822] pci 0000:00:04.0: BAR 0: assigned [mem 0xf300c000-0xf300ffff 64bit pref]
[  108.125204] pci 0000:00:04.0: BAR 3: assigned [mem 0xf3010000-0xf3013fff 64bit pref]
[  108.133651] ixgbevf 0000:00:04.0: enabling device (0000 -> 0002)
[  108.155937] ixgbevf 0000:00:04.0: 0a:b5:1f:03:d1:d6
[  108.161356] ixgbevf 0000:00:04.0: MAC: 1
[  108.165384] ixgbevf 0000:00:04.0: Intel(R) 82599 Virtual Function
[  109.207598] ixgbevf 0000:00:04.0: NIC Link is Up 10 Gbps
[  109.209123] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
[  109.216945] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready
[  113.679942] IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP)
[  113.685717] IPVS: Connection hash table configured (size=4096, memory=64Kbytes)
[  113.692556] IPVS: Creating netns size=2104 id=0
[  113.696910] IPVS: Creating netns size=2104 id=1
[  113.701795] IPVS: Creating netns size=2104 id=2
[  113.707019] IPVS: ipvs loaded.
[  113.928168] ip_set: protocol 6
[  114.518255] ctnetlink v0.93: registering with nfnetlink.
[  116.944614] eth0: renamed from veth6a26ab53
[  117.140879] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[  117.160731] IPv6: ADDRCONF(NETDEV_UP): veth3bb00fe4: link is not ready
[  117.166078] IPv6: ADDRCONF(NETDEV_CHANGE): veth3bb00fe4: link becomes ready
[  117.171070] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[  118.797087] eth0: renamed from veth866fedd4
[  118.833911] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[  118.856871] IPv6: ADDRCONF(NETDEV_UP): vethd2cdd6d5: link is not ready
[  118.864222] IPv6: ADDRCONF(NETDEV_CHANGE): vethd2cdd6d5: link becomes ready
[  118.870621] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[  118.916169] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[  123.629276] systemd[1]: apt-daily.timer: Adding 11h 50min 24.895258s random time.
[397680.813719] IPVS: Creating netns size=2104 id=3
[397682.496358] eth0: renamed from veth1c4de472
[397682.533527] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[397682.556585] IPv6: ADDRCONF(NETDEV_UP): veth511cd2e7: link is not ready
[397682.562614] IPv6: ADDRCONF(NETDEV_CHANGE): veth511cd2e7: link becomes ready
[397682.568937] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[397683.164213] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[397735.833284] ip6_tables: (C) 2000-2006 Netfilter Core Team
[398743.679352] IPVS: Creating netns size=2104 id=4
[398743.827482] IPVS: Creating netns size=2104 id=5
[398743.986820] eth0: renamed from veth80b10239
[398744.022859] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[398744.056617] IPv6: ADDRCONF(NETDEV_UP): veth07bff6ad: link is not ready
[398744.062720] IPv6: ADDRCONF(NETDEV_CHANGE): veth07bff6ad: link becomes ready
[398744.071079] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[398744.956158] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[398748.611469] eth0: renamed from veth045a018e
[398748.654889] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[398748.680686] IPv6: ADDRCONF(NETDEV_UP): vethd9f62187: link is not ready
[398748.686450] IPv6: ADDRCONF(NETDEV_CHANGE): vethd9f62187: link becomes ready
[398748.692840] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[398749.244109] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[587038.664685] IPVS: Creating netns size=2104 id=6
[587042.498485] eth0: renamed from veth3b461956
[587042.533914] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[587042.556712] IPv6: ADDRCONF(NETDEV_UP): veth9c88d69c: link is not ready
[587042.564446] IPv6: ADDRCONF(NETDEV_CHANGE): veth9c88d69c: link becomes ready
[587042.572508] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[587043.324165] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[629588.394598] IPVS: Creating netns size=2104 id=7
[629589.033645] IPVS: Creating netns size=2104 id=8
[629591.303734] eth0: renamed from veth9cc78210
[629591.341668] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[629591.364883] IPv6: ADDRCONF(NETDEV_UP): veth5cde9f27: link is not ready
[629591.368594] IPv6: ADDRCONF(NETDEV_CHANGE): veth5cde9f27: link becomes ready
[629591.373438] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[629591.504162] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[629596.834900] IPVS: Creating netns size=2104 id=9
[629597.870974] eth0: renamed from vethb0fd58e2
[629597.945446] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[629597.984895] IPv6: ADDRCONF(NETDEV_UP): veth4fcdaced: link is not ready
[629597.995886] IPv6: ADDRCONF(NETDEV_CHANGE): veth4fcdaced: link becomes ready
[629598.021694] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[629598.492175] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[629604.666020] eth0: renamed from veth1ce15a0e
[629604.722842] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[629604.768791] IPv6: ADDRCONF(NETDEV_UP): veth6359a6a1: link is not ready
[629604.775207] IPv6: ADDRCONF(NETDEV_CHANGE): veth6359a6a1: link becomes ready
[629604.789803] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[629605.084233] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[1150473.622516] IPVS: Creating netns size=2104 id=10
[1150473.795416] eth0: renamed from vetha6bc7b69
[1150473.822824] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[1150473.831547] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[1150473.856633] IPv6: ADDRCONF(NETDEV_UP): veth97ac17f3: link is not ready
[1150473.861103] IPv6: ADDRCONF(NETDEV_CHANGE): veth97ac17f3: link becomes ready
[1150473.865583] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[1219688.857840] IPVS: Creating netns size=2104 id=11
[1219689.031635] eth0: renamed from veth2d9cf107
[1219689.065596] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[1219689.088738] IPv6: ADDRCONF(NETDEV_UP): veth0bb02fab: link is not ready
[1219689.094882] IPv6: ADDRCONF(NETDEV_CHANGE): veth0bb02fab: link becomes ready
[1219689.101418] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[1219689.532152] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[1231653.006314] IPVS: Creating netns size=2104 id=12
[1231653.189635] eth0: renamed from veth79bae9d5
[1231653.249798] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[1231653.272617] IPv6: ADDRCONF(NETDEV_UP): veth0be37cd9: link is not ready
[1231653.277361] IPv6: ADDRCONF(NETDEV_CHANGE): veth0be37cd9: link becomes ready
[1231653.282262] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[1231654.048180] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[1487906.219290] IPVS: Creating netns size=2104 id=13
[1487906.301168] IPVS: Creating netns size=2104 id=14
[1487906.457485] eth0: renamed from vetheeb70d3e
[1487906.489692] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[1487906.512760] IPv6: ADDRCONF(NETDEV_UP): veth9bbe1411: link is not ready
[1487906.516833] IPv6: ADDRCONF(NETDEV_CHANGE): veth9bbe1411: link becomes ready
[1487906.521192] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[1487906.592938] eth0: renamed from veth1d0180fc
[1487906.625862] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[1487906.645057] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[1487906.712173] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[1487906.716608] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[1571831.586684] IPVS: Creating netns size=2104 id=15
[1571831.773575] eth0: renamed from veth387b3fc6
[1571831.812970] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[1571831.842950] IPv6: ADDRCONF(NETDEV_UP): vetha450bf43: link is not ready
[1571831.847003] IPv6: ADDRCONF(NETDEV_CHANGE): vetha450bf43: link becomes ready
[1571831.851036] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[1571832.796207] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[1934566.860403] IPVS: Creating netns size=2104 id=16
[1934567.020610] eth0: renamed from veth8229c58b
[1934567.057648] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[1934567.080587] IPv6: ADDRCONF(NETDEV_UP): veth5c65b329: link is not ready
[1934567.085834] IPv6: ADDRCONF(NETDEV_CHANGE): veth5c65b329: link becomes ready
[1934567.091871] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[1934567.514474] IPVS: Creating netns size=2104 id=17
[1934567.616241] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[1934567.707595] eth0: renamed from vethbe4d5d58
[1934567.741476] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[1934567.768845] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[1934568.700288] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[1934568.884984] IPVS: Creating netns size=2104 id=18
[1934569.269082] IPVS: Creating netns size=2104 id=19
[1934573.696555] eth0: renamed from veth8364279d
[1934573.744135] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[1934573.773081] IPv6: ADDRCONF(NETDEV_UP): veth0226b36a: link is not ready
[1934573.784856] IPv6: ADDRCONF(NETDEV_CHANGE): veth0226b36a: link becomes ready
[1934573.793294] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[1934574.016247] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[1934580.428703] eth0: renamed from vethefcc297b
[1934580.471455] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[1934580.496526] IPv6: ADDRCONF(NETDEV_UP): veth6876a073: link is not ready
[1934580.502127] IPv6: ADDRCONF(NETDEV_CHANGE): veth6876a073: link becomes ready
[1934580.508067] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[1934581.436197] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[2245282.765847] IPVS: Creating netns size=2104 id=20
[2245287.592183] eth0: renamed from veth7514b94b
[2245287.634481] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[2245287.660846] IPv6: ADDRCONF(NETDEV_UP): veth78537dd3: link is not ready
[2245287.664982] IPv6: ADDRCONF(NETDEV_CHANGE): veth78537dd3: link becomes ready
[2245287.669672] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[2245287.832173] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[2613454.020175] hrtimer: interrupt took 23271 ns
[2758140.734421] IPVS: Creating netns size=2104 id=21
[2758144.571305] eth0: renamed from veth5a8f463b
[2758144.613933] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[2758144.636703] IPv6: ADDRCONF(NETDEV_UP): vetha89df936: link is not ready
[2758144.644328] IPv6: ADDRCONF(NETDEV_CHANGE): vetha89df936: link becomes ready
[2758144.652746] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[2758145.020239] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[2777454.099727] IPVS: Creating netns size=2104 id=22
[2777457.926835] eth0: renamed from vetha46bd121
[2777457.961649] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[2777457.968525] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[2777457.996464] IPv6: ADDRCONF(NETDEV_UP): veth20c20e42: link is not ready
[2777458.003461] IPv6: ADDRCONF(NETDEV_CHANGE): veth20c20e42: link becomes ready
[2777458.010896] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[2777714.900729] IPVS: Creating netns size=2104 id=23
[2777716.572992] eth0: renamed from veth1a349b65
[2777716.610173] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[2777716.636750] IPv6: ADDRCONF(NETDEV_UP): veth9694e96f: link is not ready
[2777716.642911] IPv6: ADDRCONF(NETDEV_CHANGE): veth9694e96f: link becomes ready
[2777716.649732] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[2777716.848285] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[3048892.048849] IPVS: Creating netns size=2104 id=24
[3048899.023850] eth0: renamed from veth3b260040
[3048899.053771] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[3048899.076600] IPv6: ADDRCONF(NETDEV_UP): veth49dfca24: link is not ready
[3048899.082654] IPv6: ADDRCONF(NETDEV_CHANGE): veth49dfca24: link becomes ready
[3048899.089102] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[3048899.676258] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[5265983.939285] IPVS: Creating netns size=2104 id=25
[5265984.024598] IPVS: Creating netns size=2104 id=26
[5265984.223797] eth0: renamed from vethcdd19c83
[5265984.262239] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[5265984.284596] IPv6: ADDRCONF(NETDEV_UP): veth79722822: link is not ready
[5265984.290269] IPv6: ADDRCONF(NETDEV_CHANGE): veth79722822: link becomes ready
[5265984.294990] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[5265984.397680] eth0: renamed from vethd841126b
[5265984.442479] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[5265984.464903] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[5265984.764224] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[5265984.769872] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[5266158.145313] IPVS: Creating netns size=2104 id=27
[5266158.367780] eth0: renamed from veth463c94da
[5266158.434340] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[5266158.460566] IPv6: ADDRCONF(NETDEV_UP): veth4e97c996: link is not ready
[5266158.468092] IPv6: ADDRCONF(NETDEV_CHANGE): veth4e97c996: link becomes ready
[5266158.475842] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[5266159.392284] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[5267209.827696] IPVS: Creating netns size=2104 id=28
[5267210.008989] IPVS: Creating netns size=2104 id=29
[5267210.196922] eth0: renamed from vethe684230b
[5267210.233912] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[5267210.260728] IPv6: ADDRCONF(NETDEV_UP): veth131f488b: link is not ready
[5267210.264656] IPv6: ADDRCONF(NETDEV_CHANGE): veth131f488b: link becomes ready
[5267210.269011] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[5267210.424365] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[5267215.903723] pci 0000:00:05.0: [8086:10ed] type 00 class 0x020000
[5267215.905254] pci 0000:00:05.0: reg 0x10: [mem 0x00000000-0x00003fff 64bit pref]
[5267215.906486] pci 0000:00:05.0: reg 0x1c: [mem 0x00000000-0x00003fff 64bit pref]
[5267215.909702] pci 0000:00:05.0: BAR 0: assigned [mem 0xf3014000-0xf3017fff 64bit pref]
[5267215.942073] pci 0000:00:05.0: BAR 3: assigned [mem 0xf3018000-0xf301bfff 64bit pref]
[5267215.982156] ixgbevf 0000:00:05.0: enabling device (0000 -> 0002)
[5267216.006188] ixgbevf 0000:00:05.0: 0a:e2:0f:b4:49:4c
[5267216.011425] ixgbevf 0000:00:05.0: MAC: 1
[5267216.014726] ixgbevf 0000:00:05.0: Intel(R) 82599 Virtual Function
[5267217.066780] ixgbevf 0000:00:05.0: NIC Link is Up 10 Gbps
[5267217.068302] IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready
[5267217.077862] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
[5267223.760033] eth0: renamed from veth73944129
[5267223.811241] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[5267223.840565] IPv6: ADDRCONF(NETDEV_UP): vetheecbc914: link is not ready
[5267223.846126] IPv6: ADDRCONF(NETDEV_CHANGE): vetheecbc914: link becomes ready
[5267223.852078] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[5267515.408411] IPVS: Creating netns size=2104 id=30
[5267515.551943] IPVS: Creating netns size=2104 id=31
[5267516.017727] IPVS: Creating netns size=2104 id=32
[5267516.635825] NOHZ: local_softirq_pending 08
[5267519.600275] eth0: renamed from veth633c70ac
[5267519.683885] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[5267519.721773] IPv6: ADDRCONF(NETDEV_UP): vethd7da3bb3: link is not ready
[5267519.728327] IPv6: ADDRCONF(NETDEV_CHANGE): vethd7da3bb3: link becomes ready
[5267519.734532] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[5267520.540135] IPv6: eth0: IPv6 duplicate address fe80::8e2:fff:feb4:494c detected!
[5267520.711517] NOHZ: local_softirq_pending 08
[5267525.214398] eth0: renamed from vethc68cba8b
[5267525.266369] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[5267525.317337] IPv6: ADDRCONF(NETDEV_UP): vethdc042b66: link is not ready
[5267525.337959] IPv6: ADDRCONF(NETDEV_CHANGE): vethdc042b66: link becomes ready
[5267525.354880] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[5267525.692178] IPv6: eth0: IPv6 duplicate address fe80::8e2:fff:feb4:494c detected!
[5267526.804782] NOHZ: local_softirq_pending 08
[5267530.144381] eth0: renamed from vethab835d14
[5267530.183442] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[5267530.213012] IPv6: ADDRCONF(NETDEV_UP): veth76292a29: link is not ready
[5267530.222442] IPv6: ADDRCONF(NETDEV_CHANGE): veth76292a29: link becomes ready
[5267530.229198] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[5267530.784177] IPv6: eth0: IPv6 duplicate address fe80::8e2:fff:feb4:494c detected!
[5287044.336438] IPVS: Creating netns size=2104 id=33
[5287045.502358] NOHZ: local_softirq_pending 08
[5287047.262368] eth0: renamed from vethe25baad9
[5287047.298218] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[5287047.324802] IPv6: ADDRCONF(NETDEV_UP): veth2c3fe2c9: link is not ready
[5287047.331833] IPv6: ADDRCONF(NETDEV_CHANGE): veth2c3fe2c9: link becomes ready
[5287047.339177] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[5287047.996172] IPv6: eth0: IPv6 duplicate address fe80::8e2:fff:feb4:494c detected!
[5688408.752170] updating oom_score_adj for 30911 (kubelet) from -999 to -999 because it shares mm with 1805 (kubelet). Report if this is unexpected.
[7073046.915035] IPVS: Creating netns size=2104 id=34
[7073047.330575] IPVS: Creating netns size=2104 id=35
[7073047.510905] IPVS: Creating netns size=2104 id=36
[7073047.698762] IPVS: Creating netns size=2104 id=37
[7073048.438218] NOHZ: local_softirq_pending 08
[7073050.903121] eth0: renamed from vethd1c35254
[7073050.942829] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[7073050.973263] IPv6: ADDRCONF(NETDEV_UP): veth0d6500b2: link is not ready
[7073050.980126] IPv6: ADDRCONF(NETDEV_CHANGE): veth0d6500b2: link becomes ready
[7073050.987111] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[7073051.076221] IPv6: eth0: IPv6 duplicate address fe80::8e2:fff:feb4:494c detected!
[7073052.480226] NOHZ: local_softirq_pending 08
[7073054.542658] eth0: renamed from veth438f2404
[7073054.579297] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[7073054.604759] IPv6: ADDRCONF(NETDEV_UP): veth6cf11f6f: link is not ready
[7073054.612882] IPv6: ADDRCONF(NETDEV_CHANGE): veth6cf11f6f: link becomes ready
[7073054.621063] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[7073054.972282] IPv6: eth0: IPv6 duplicate address fe80::8e2:fff:feb4:494c detected!
[7073055.565302] NOHZ: local_softirq_pending 08
[7073059.192723] eth0: renamed from veth3a4a5fc6
[7073059.247501] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[7073059.288424] IPv6: ADDRCONF(NETDEV_UP): veth7ed9d4a2: link is not ready
[7073059.295011] IPv6: ADDRCONF(NETDEV_CHANGE): veth7ed9d4a2: link becomes ready
[7073059.301383] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[7073059.872207] IPv6: eth0: IPv6 duplicate address fe80::8e2:fff:feb4:494c detected!
[7073060.629208] NOHZ: local_softirq_pending 08
[7073066.036456] eth0: renamed from veth0e22813b
[7073066.083210] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[7073066.125565] IPv6: ADDRCONF(NETDEV_UP): veth85cf11f4: link is not ready
[7073066.140149] IPv6: ADDRCONF(NETDEV_CHANGE): veth85cf11f4: link becomes ready
[7073066.146345] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[7073066.684215] IPv6: eth0: IPv6 duplicate address fe80::8e2:fff:feb4:494c detected!
[8119806.701095] IPVS: Creating netns size=2104 id=38
[8119807.037502] eth0: renamed from veth1edfa2ac
[8119807.079161] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[8119807.117661] IPv6: ADDRCONF(NETDEV_UP): veth6d26d70f: link is not ready
[8119807.125073] IPv6: ADDRCONF(NETDEV_CHANGE): veth6d26d70f: link becomes ready
[8119807.132665] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[8119807.612224] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[8192798.151330] systemd: 35 output lines suppressed due to ratelimiting
[9705236.821291] IPVS: Creating netns size=2104 id=39
[9705237.122211] eth0: renamed from veth3bc908a6
[9705237.173377] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[9705237.194916] IPv6: ADDRCONF(NETDEV_UP): vethd6d10032: link is not ready
[9705237.200406] IPv6: ADDRCONF(NETDEV_CHANGE): vethd6d10032: link becomes ready
[9705237.205863] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[9705237.352258] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[10014269.750032] IPVS: Creating netns size=2104 id=40
[10014270.089951] eth0: renamed from veth72e61638
[10014270.128167] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[10014270.161642] IPv6: ADDRCONF(NETDEV_UP): veth2274e19b: link is not ready
[10014270.166598] IPv6: ADDRCONF(NETDEV_CHANGE): veth2274e19b: link becomes ready
[10014270.172405] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[10014270.812292] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[10014998.826841] IPVS: Creating netns size=2104 id=41
[10014999.130951] eth0: renamed from veth379e2ae9
[10014999.182432] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[10014999.218279] IPv6: ADDRCONF(NETDEV_UP): veth5af52012: link is not ready
[10014999.224901] IPv6: ADDRCONF(NETDEV_CHANGE): veth5af52012: link becomes ready
[10014999.231783] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[10014999.772252] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[10029116.113264] IPVS: Creating netns size=2104 id=42
[10029116.315615] IPVS: Creating netns size=2104 id=43
[10029116.689761] eth0: renamed from vethb35d53bf
[10029116.728847] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[10029116.762888] IPv6: ADDRCONF(NETDEV_UP): vethc804ec7d: link is not ready
[10029116.767250] IPv6: ADDRCONF(NETDEV_CHANGE): vethc804ec7d: link becomes ready
[10029116.772204] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[10029116.973413] eth0: renamed from veth471c751d
[10029117.014556] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[10029117.043396] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[10029117.468246] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[10029117.475092] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[10029174.085999] IPVS: Creating netns size=2104 id=44
[10029174.609804] eth0: renamed from vethd80eda38
[10029174.664374] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[10029174.696860] IPv6: ADDRCONF(NETDEV_UP): veth9327937b: link is not ready
[10029174.712134] IPv6: ADDRCONF(NETDEV_CHANGE): veth9327937b: link becomes ready
[10029174.717794] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[10029174.876294] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[10029177.977548] IPVS: Creating netns size=2104 id=45
[10029182.609901] eth0: renamed from vetha9aa6571
[10029182.672624] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[10029182.704959] IPv6: ADDRCONF(NETDEV_UP): veth5ae9c7f7: link is not ready
[10029182.710686] IPv6: ADDRCONF(NETDEV_CHANGE): veth5ae9c7f7: link becomes ready
[10029182.716963] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[10029183.036257] IPv6: eth0: IPv6 duplicate address fe80::8e2:fff:feb4:494c detected!
[10029557.975711] IPVS: Creating netns size=2104 id=46
[10029558.335964] eth0: renamed from vethcd384687
[10029558.383079] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[10029558.417562] IPv6: ADDRCONF(NETDEV_UP): vethe90517b3: link is not ready
[10029558.425822] IPv6: ADDRCONF(NETDEV_CHANGE): vethe90517b3: link becomes ready
[10029558.433014] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[10029559.388310] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[10031980.455002] IPVS: Creating netns size=2104 id=47
[10031980.762450] eth0: renamed from vethab63bcd4
[10031980.815315] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[10031980.840909] IPv6: ADDRCONF(NETDEV_UP): veth29f12c09: link is not ready
[10031980.847492] IPv6: ADDRCONF(NETDEV_CHANGE): veth29f12c09: link becomes ready
[10031980.853841] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[10031981.628217] IPv6: eth0: IPv6 duplicate address fe80::8e2:fff:feb4:494c detected!
[10032049.748510] IPVS: Creating netns size=2104 id=48
[10032051.718535] eth0: renamed from veth19ae7009
[10032051.754887] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[10032051.784689] IPv6: ADDRCONF(NETDEV_UP): veth88cbb1f9: link is not ready
[10032051.790495] IPv6: ADDRCONF(NETDEV_CHANGE): veth88cbb1f9: link becomes ready
[10032051.796826] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[10032052.060201] IPv6: eth0: IPv6 duplicate address fe80::8e2:fff:feb4:494c detected!
[10032662.576526] IPVS: Creating netns size=2104 id=49
[10032662.882307] eth0: renamed from veth96669347
[10032662.914826] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[10032662.941092] IPv6: ADDRCONF(NETDEV_UP): vethffb0d786: link is not ready
[10032662.946405] IPv6: ADDRCONF(NETDEV_CHANGE): vethffb0d786: link becomes ready
[10032662.951894] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[10032663.056271] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[10033210.489497] IPVS: Creating netns size=2104 id=50
[10033210.736545] eth0: renamed from veth01585891
[10033210.777965] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[10033210.800866] IPv6: ADDRCONF(NETDEV_UP): veth7d5911f5: link is not ready
[10033210.804867] IPv6: ADDRCONF(NETDEV_CHANGE): veth7d5911f5: link becomes ready
[10033210.809409] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[10033210.840267] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[10033559.520207] IPVS: Creating netns size=2104 id=51
[10033559.808861] eth0: renamed from veth56e8dbb3
[10033559.855364] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[10033559.881362] IPv6: ADDRCONF(NETDEV_UP): vethe35eb799: link is not ready
[10033559.887641] IPv6: ADDRCONF(NETDEV_CHANGE): vethe35eb799: link becomes ready
[10033559.894144] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[10033560.220223] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[10033561.307110] IPVS: Creating netns size=2104 id=52
[10033561.552057] eth0: renamed from vetha04f241c
[10033561.598814] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[10033561.633843] IPv6: ADDRCONF(NETDEV_UP): vethb3a35b18: link is not ready
[10033561.638124] IPv6: ADDRCONF(NETDEV_CHANGE): vethb3a35b18: link becomes ready
[10033561.642835] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[10033562.364329] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[10033570.672364] IPVS: Creating netns size=2104 id=53
[10033570.919380] eth0: renamed from vetha4bf36b4
[10033570.966305] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[10033570.997794] IPv6: ADDRCONF(NETDEV_UP): vetheab9e924: link is not ready
[10033571.003278] IPv6: ADDRCONF(NETDEV_CHANGE): vetheab9e924: link becomes ready
[10033571.009423] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[10033571.477268] IPVS: Creating netns size=2104 id=54
[10033571.503009] IPVS: Creating netns size=2104 id=55
[10033571.779513] eth0: renamed from veth8f8b5698
[10033571.825071] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[10033571.854678] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[10033571.964341] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[10033572.000010] eth0: renamed from veth2bf394ac
[10033572.055852] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[10033572.084617] IPv6: ADDRCONF(NETDEV_UP): vethc5a5706b: link is not ready
[10033572.091170] IPv6: ADDRCONF(NETDEV_CHANGE): vethc5a5706b: link becomes ready
[10033572.097960] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[10033572.572152] IPv6: eth0: IPv6 duplicate address fe80::8e2:fff:feb4:494c detected!
[10034134.453811] IPVS: Creating netns size=2104 id=56
[10034134.764485] eth0: renamed from veth4329adfc
[10034134.810107] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[10034134.845404] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[10034135.484335] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[10034147.193857] IPVS: Creating netns size=2104 id=57
[10034147.568644] eth0: renamed from vethed5b435c
[10034147.607781] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[10034147.641447] IPv6: ADDRCONF(NETDEV_UP): veth0758d8b9: link is not ready
[10034147.647124] IPv6: ADDRCONF(NETDEV_CHANGE): veth0758d8b9: link becomes ready
[10034147.653019] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[10034148.448182] IPv6: eth0: IPv6 duplicate address fe80::8e2:fff:feb4:494c detected!
[10035216.473923] IPVS: Creating netns size=2104 id=58
[10035216.786377] eth0: renamed from vethaeb5efc0
[10035216.823415] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[10035216.852972] IPv6: ADDRCONF(NETDEV_UP): vethf959f696: link is not ready
[10035216.857744] IPv6: ADDRCONF(NETDEV_CHANGE): vethf959f696: link becomes ready
[10035216.863785] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[10035217.436298] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[10734678.004518] IPVS: Creating netns size=2104 id=59
[10734679.046803] eth0: renamed from veth992cff0e
[10734679.149999] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[10734679.180876] IPv6: ADDRCONF(NETDEV_UP): veth0e6693b3: link is not ready
[10734679.187790] IPv6: ADDRCONF(NETDEV_CHANGE): veth0e6693b3: link becomes ready
[10734679.194874] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[10734679.484293] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[10898051.273149] IPVS: Creating netns size=2104 id=60
[10898051.856433] eth0: renamed from veth8511e8a5
[10898051.996473] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[10898052.025941] IPv6: ADDRCONF(NETDEV_UP): veth8e7c7aea: link is not ready
[10898052.034454] IPv6: ADDRCONF(NETDEV_CHANGE): veth8e7c7aea: link becomes ready
[10898052.042090] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[10898052.192249] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[11154433.052297] IPVS: Creating netns size=2104 id=61
[11154433.813635] eth0: renamed from veth99bf870f
[11154433.896994] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[11154433.916600] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[11154433.917755] IPv6: ADDRCONF(NETDEV_UP): veth2ea58c24: link is not ready
[11154433.918407] IPv6: ADDRCONF(NETDEV_CHANGE): veth2ea58c24: link becomes ready
[11154433.918491] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[11321369.821849] IPVS: Creating netns size=2104 id=62
[11321370.244558] eth0: renamed from vethe8e9cb16
[11321370.298176] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[11321370.345591] IPv6: ADDRCONF(NETDEV_UP): vethe8bbb114: link is not ready
[11321370.352895] IPv6: ADDRCONF(NETDEV_CHANGE): vethe8bbb114: link becomes ready
[11321370.360388] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[11321371.100251] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[13557181.011235] IPVS: Creating netns size=2104 id=63
[13557181.688122] eth0: renamed from vethfd3057fe
[13557181.752213] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[13557181.760950] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[13557181.786568] IPv6: ADDRCONF(NETDEV_UP): vethb780b5f8: link is not ready
[13557181.791260] IPv6: ADDRCONF(NETDEV_CHANGE): vethb780b5f8: link becomes ready
[13557181.795899] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[13557199.931529] IPVS: Creating netns size=2104 id=64
[13557200.163640] eth0: renamed from veth89d1e62e
[13557200.213197] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[13557200.244733] IPv6: ADDRCONF(NETDEV_UP): veth785bdef0: link is not ready
[13557200.250614] IPv6: ADDRCONF(NETDEV_CHANGE): veth785bdef0: link becomes ready
[13557200.256684] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[13557200.480317] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[14815285.791558] IPVS: Creating netns size=2104 id=65
[14815286.666192] eth0: renamed from vethf2e569d3
[14815286.792285] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[14815286.825189] IPv6: ADDRCONF(NETDEV_UP): veth44644909: link is not ready
[14815286.828971] IPv6: ADDRCONF(NETDEV_CHANGE): veth44644909: link becomes ready
[14815286.833466] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[14815287.356229] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[14819626.198971] IPVS: Creating netns size=2104 id=66
[14819626.440706] eth0: renamed from veth4c803290
[14819626.473657] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[14819626.496612] IPv6: ADDRCONF(NETDEV_UP): veth4a09c4c8: link is not ready
[14819626.501104] IPv6: ADDRCONF(NETDEV_CHANGE): veth4a09c4c8: link becomes ready
[14819626.505481] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[14819626.876278] IPv6: eth0: IPv6 duplicate address fe80::8e2:fff:feb4:494c detected!
[14854155.134087] IPVS: Creating netns size=2104 id=67
[14854155.419212] eth0: renamed from veth4f1df856
[14854155.454942] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[14854155.476968] IPv6: ADDRCONF(NETDEV_UP): vethb4ff36c3: link is not ready
[14854155.483177] IPv6: ADDRCONF(NETDEV_CHANGE): vethb4ff36c3: link becomes ready
[14854155.489323] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[14854155.580305] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[14902926.879984] IPVS: Creating netns size=2104 id=68
[14902927.146955] eth0: renamed from vethe095c60d
[14902927.198395] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[14902927.216735] IPv6: ADDRCONF(NETDEV_UP): veth86c8571a: link is not ready
[14902927.221995] IPv6: ADDRCONF(NETDEV_CHANGE): veth86c8571a: link becomes ready
[14902927.227935] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[14902927.964255] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[15016737.805971] IPVS: Creating netns size=2104 id=69
[15016739.212357] eth0: renamed from veth8f165585
[15016739.296169] IPv6: eth0: IPv6 duplicate address fe80::8e2:fff:feb4:494c detected!
[15016739.323720] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[15016739.362729] IPv6: ADDRCONF(NETDEV_UP): vethc2cd355b: link is not ready
[15016739.367962] IPv6: ADDRCONF(NETDEV_CHANGE): vethc2cd355b: link becomes ready
[15016739.372931] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[15019006.638635] IPVS: Creating netns size=2104 id=70
[15019007.754074] eth0: renamed from vethfa6c463f
[15019007.853050] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[15019007.893505] IPv6: ADDRCONF(NETDEV_UP): veth497dec3c: link is not ready
[15019007.908485] IPv6: ADDRCONF(NETDEV_CHANGE): veth497dec3c: link becomes ready
[15019007.918452] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[15019008.668203] IPv6: eth0: IPv6 duplicate address fe80::8e2:fff:feb4:494c detected!
[15019762.852537] IPVS: Creating netns size=2104 id=71
[15019763.077725] eth0: renamed from veth302e1529
[15019763.130428] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[15019763.152995] IPv6: ADDRCONF(NETDEV_UP): veth06b6951f: link is not ready
[15019763.158629] IPv6: ADDRCONF(NETDEV_CHANGE): veth06b6951f: link becomes ready
[15019763.165057] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[15019763.772268] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[15019805.967288] IPVS: Creating netns size=2104 id=72
[15019806.216381] eth0: renamed from vetha19cf3df
[15019806.258605] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[15019806.284648] IPv6: ADDRCONF(NETDEV_UP): veth8bf991eb: link is not ready
[15019806.292567] IPv6: ADDRCONF(NETDEV_CHANGE): veth8bf991eb: link becomes ready
[15019806.297774] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[15019806.620193] IPv6: eth0: IPv6 duplicate address fe80::8e2:fff:feb4:494c detected!
[16293431.185210] updating oom_score_adj for 7246 (kubelet) from -999 to -999 because it shares mm with 1805 (kubelet). Report if this is unexpected.
[19616762.935323] IPVS: Creating netns size=2104 id=73
[19616763.317700] eth0: renamed from veth33d34fda
[19616763.368203] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[19616763.400740] IPv6: ADDRCONF(NETDEV_UP): veth2c842b79: link is not ready
[19616763.406227] IPv6: ADDRCONF(NETDEV_CHANGE): veth2c842b79: link becomes ready
[19616763.411863] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[19616763.432259] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[19619690.906349] IPVS: Creating netns size=2104 id=74
[19619691.160011] eth0: renamed from veth257bc40e
[19619691.199135] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[19619691.224812] IPv6: ADDRCONF(NETDEV_UP): vetha902fc21: link is not ready
[19619691.231187] IPv6: ADDRCONF(NETDEV_CHANGE): vetha902fc21: link becomes ready
[19619691.236291] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[19619691.248318] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[19621208.663113] IPVS: Creating netns size=2104 id=75
[19621208.940140] eth0: renamed from vethf109b1b0
[19621208.973797] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[19621209.004753] IPv6: ADDRCONF(NETDEV_UP): vethf685a576: link is not ready
[19621209.009703] IPv6: ADDRCONF(NETDEV_CHANGE): vethf685a576: link becomes ready
[19621209.015208] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[19621209.724271] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[19621775.005892] IPVS: Creating netns size=2104 id=76
[19621775.229799] eth0: renamed from vethe1f24f53
[19621775.286527] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[19621775.312729] IPv6: ADDRCONF(NETDEV_UP): vethd74d2147: link is not ready
[19621775.318395] IPv6: ADDRCONF(NETDEV_CHANGE): vethd74d2147: link becomes ready
[19621775.325050] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[19621775.932269] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[19621778.379133] IPVS: Creating netns size=2104 id=77
[19621778.658998] eth0: renamed from veth378c6c21
[19621778.702653] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[19621778.740936] IPv6: ADDRCONF(NETDEV_UP): veth28ea9fa6: link is not ready
[19621778.747456] IPv6: ADDRCONF(NETDEV_CHANGE): veth28ea9fa6: link becomes ready
[19621778.754344] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[19621779.228260] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[19622388.509144] IPVS: Creating netns size=2104 id=78
[19622388.760501] eth0: renamed from veth3e1e5af8
[19622388.805969] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[19622388.836562] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[19622388.836706] IPv6: ADDRCONF(NETDEV_UP): vethcae6268a: link is not ready
[19622388.836716] IPv6: ADDRCONF(NETDEV_CHANGE): vethcae6268a: link becomes ready
[19622388.836823] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[19690945.838812] IPVS: Creating netns size=2104 id=79
[19690946.075372] eth0: renamed from veth8992893c
[19690946.109768] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[19690946.136719] IPv6: ADDRCONF(NETDEV_UP): veth99a30974: link is not ready
[19690946.142270] IPv6: ADDRCONF(NETDEV_CHANGE): veth99a30974: link becomes ready
[19690946.148227] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[19690946.556324] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[19694753.362196] IPVS: Creating netns size=2104 id=80
[19694753.578352] eth0: renamed from veth837d8c02
[19694753.613754] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[19694753.644858] IPv6: ADDRCONF(NETDEV_UP): vethb5b6b69b: link is not ready
[19694753.650571] IPv6: ADDRCONF(NETDEV_CHANGE): vethb5b6b69b: link becomes ready
[19694753.656310] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[19694753.916286] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[19694942.785845] IPVS: Creating netns size=2104 id=81
[19694943.000559] eth0: renamed from veth6706eaef
[19694943.057675] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[19694943.092696] IPv6: ADDRCONF(NETDEV_UP): veth077d00c2: link is not ready
[19694943.098067] IPv6: ADDRCONF(NETDEV_CHANGE): veth077d00c2: link becomes ready
[19694943.103552] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[19694943.868283] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[19695077.140411] IPVS: Creating netns size=2104 id=82
[19695077.377371] eth0: renamed from veth61c57b8e
[19695077.425295] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[19695077.452806] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[19695078.172338] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[19695181.815157] IPVS: Creating netns size=2104 id=83
[19695182.051353] eth0: renamed from veth1587e0a3
[19695182.081492] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[19695182.104998] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[19695182.844341] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[19695437.997207] IPVS: Creating netns size=2104 id=84
[19695438.250284] eth0: renamed from vethe54eed3a
[19695438.290308] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[19695438.313176] IPv6: ADDRCONF(NETDEV_UP): veth6eb13d04: link is not ready
[19695438.317776] IPv6: ADDRCONF(NETDEV_CHANGE): veth6eb13d04: link becomes ready
[19695438.323027] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[19695438.720337] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[19696438.850029] IPVS: Creating netns size=2104 id=85
[19696439.064595] eth0: renamed from veth51432c18
[19696439.113677] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[19696439.140747] IPv6: ADDRCONF(NETDEV_UP): veth7e357f1f: link is not ready
[19696439.146663] IPv6: ADDRCONF(NETDEV_CHANGE): veth7e357f1f: link becomes ready
[19696439.152901] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[19696439.740287] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[19697163.050061] IPVS: Creating netns size=2104 id=86
[19697163.279406] eth0: renamed from veth563d3808
[19697163.327167] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[19697163.349054] IPv6: ADDRCONF(NETDEV_UP): veth62aa98b2: link is not ready
[19697163.354890] IPv6: ADDRCONF(NETDEV_CHANGE): veth62aa98b2: link becomes ready
[19697163.360572] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[19697163.740313] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[19697486.281133] IPVS: Creating netns size=2104 id=87
[19697486.502872] eth0: renamed from veth1a1ac29d
[19697486.537354] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[19697486.564685] IPv6: ADDRCONF(NETDEV_UP): vetheeed3c7d: link is not ready
[19697486.569860] IPv6: ADDRCONF(NETDEV_CHANGE): vetheeed3c7d: link becomes ready
[19697486.575805] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[19697486.812330] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[19699146.859741] IPVS: Creating netns size=2104 id=88
[19699147.088098] eth0: renamed from veth9db2763a
[19699147.130754] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[19699147.176580] IPv6: ADDRCONF(NETDEV_UP): veth6a50bac4: link is not ready
[19699147.184805] IPv6: ADDRCONF(NETDEV_CHANGE): veth6a50bac4: link becomes ready
[19699147.190758] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[19699147.356347] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[19699183.855409] IPVS: Creating netns size=2104 id=89
[19699184.077810] eth0: renamed from veth23a53e01
[19699184.113463] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[19699184.144631] IPv6: ADDRCONF(NETDEV_UP): veth54b1ef39: link is not ready
[19699184.150149] IPv6: ADDRCONF(NETDEV_CHANGE): veth54b1ef39: link becomes ready
[19699184.155932] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[19699184.988296] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[19699580.606724] IPVS: Creating netns size=2104 id=90
[19699580.824907] eth0: renamed from vethcd002a44
[19699580.869837] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[19699580.896765] IPv6: ADDRCONF(NETDEV_UP): veth202dea19: link is not ready
[19699580.901757] IPv6: ADDRCONF(NETDEV_CHANGE): veth202dea19: link becomes ready
[19699580.907241] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[19699581.372266] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[19700149.778835] IPVS: Creating netns size=2104 id=91
[19700150.008116] eth0: renamed from vethcaacb962
[19700150.045987] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[19700150.080803] IPv6: ADDRCONF(NETDEV_UP): vetha33099b6: link is not ready
[19700150.087597] IPv6: ADDRCONF(NETDEV_CHANGE): vetha33099b6: link becomes ready
[19700150.094486] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[19700150.200268] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[19700407.625245] IPVS: Creating netns size=2104 id=92
[19700407.868481] eth0: renamed from vethdada585a
[19700407.926006] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[19700407.952796] IPv6: ADDRCONF(NETDEV_UP): veth4bdc34ec: link is not ready
[19700407.958430] IPv6: ADDRCONF(NETDEV_CHANGE): veth4bdc34ec: link becomes ready
[19700407.964258] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[19700408.044291] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[19700924.375291] IPVS: Creating netns size=2104 id=93
[19700924.443934] IPVS: Creating netns size=2104 id=94
[19700924.645000] eth0: renamed from veth2b74f081
[19700924.685694] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[19700924.708990] IPv6: ADDRCONF(NETDEV_UP): veth530ca844: link is not ready
[19700924.714910] IPv6: ADDRCONF(NETDEV_CHANGE): veth530ca844: link becomes ready
[19700924.721952] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[19700924.812909] eth0: renamed from veth2edf267b
[19700924.849945] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[19700924.884801] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[19700925.244231] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[19700925.252045] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[19701365.920435] IPVS: Creating netns size=2104 id=95
[19701366.162631] eth0: renamed from veth78f9c0f8
[19701366.210148] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[19701366.236916] IPv6: ADDRCONF(NETDEV_UP): veth5abf2aeb: link is not ready
[19701366.244031] IPv6: ADDRCONF(NETDEV_CHANGE): veth5abf2aeb: link becomes ready
[19701366.251098] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[19701366.352269] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[19701777.983534] IPVS: Creating netns size=2104 id=96
[19701778.204717] eth0: renamed from veth6e86cfc6
[19701778.247266] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[19701778.272681] IPv6: ADDRCONF(NETDEV_UP): veth0f26c355: link is not ready
[19701778.277740] IPv6: ADDRCONF(NETDEV_CHANGE): veth0f26c355: link becomes ready
[19701778.283270] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[19701778.716327] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[19703671.439473] IPVS: Creating netns size=2104 id=97
[19703671.656396] eth0: renamed from veth84046905
[19703671.694120] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[19703671.720925] IPv6: ADDRCONF(NETDEV_UP): veth2d7acbb0: link is not ready
[19703671.727719] IPv6: ADDRCONF(NETDEV_CHANGE): veth2d7acbb0: link becomes ready
[19703671.734649] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[19703672.572294] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[19708042.790581] IPVS: Creating netns size=2104 id=98
[19708043.019584] eth0: renamed from vethda2fc09c
[19708043.053500] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[19708043.076737] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[19708043.077038] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[19708246.032826] IPVS: Creating netns size=2104 id=99
[19708246.234419] eth0: renamed from veth438987d4
[19708246.269552] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[19708246.300777] IPv6: ADDRCONF(NETDEV_UP): veth11ac6cbb: link is not ready
[19708246.304958] IPv6: ADDRCONF(NETDEV_CHANGE): veth11ac6cbb: link becomes ready
[19708246.311078] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[19708246.656372] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[19708936.556903] IPVS: Creating netns size=2104 id=100
[19708936.787543] eth0: renamed from veth925063ef
[19708936.834657] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[19708936.860822] IPv6: ADDRCONF(NETDEV_UP): veth138c5d32: link is not ready
[19708936.864724] IPv6: ADDRCONF(NETDEV_CHANGE): veth138c5d32: link becomes ready
[19708936.868891] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[19708937.824391] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[19709656.965143] IPVS: Creating netns size=2104 id=101
[19709657.210535] eth0: renamed from vethde170bf8
[19709657.258297] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[19709657.280590] IPv6: ADDRCONF(NETDEV_UP): veth04494925: link is not ready
[19709657.285222] IPv6: ADDRCONF(NETDEV_CHANGE): veth04494925: link becomes ready
[19709657.290186] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[19709658.176271] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[19710463.816520] IPVS: Creating netns size=2104 id=102
[19710464.018813] eth0: renamed from vethe4641cb3
[19710464.049945] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[19710464.084932] IPv6: ADDRCONF(NETDEV_UP): veth646b072e: link is not ready
[19710464.089702] IPv6: ADDRCONF(NETDEV_CHANGE): veth646b072e: link becomes ready
[19710464.096043] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[19710464.636303] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[19710825.388519] IPVS: Creating netns size=2104 id=103
[19710825.627420] eth0: renamed from veth3e5a3157
[19710825.669914] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[19710825.692628] IPv6: ADDRCONF(NETDEV_UP): veth888116aa: link is not ready
[19710825.697171] IPv6: ADDRCONF(NETDEV_CHANGE): veth888116aa: link becomes ready
[19710825.702386] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[19710825.948290] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[19710919.520378] IPVS: Creating netns size=2104 id=104
[19710919.738095] eth0: renamed from vethe6549269
[19710919.777722] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[19710919.804550] IPv6: ADDRCONF(NETDEV_UP): vethd22cb6c6: link is not ready
[19710919.810179] IPv6: ADDRCONF(NETDEV_CHANGE): vethd22cb6c6: link becomes ready
[19710919.816133] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[19710920.092279] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[19712961.208763] docker0: port 1(vethdfb7acf) entered blocking state
[19712961.215127] docker0: port 1(vethdfb7acf) entered disabled state
[19712961.221659] device vethdfb7acf entered promiscuous mode
[19712961.227769] IPv6: ADDRCONF(NETDEV_UP): vethdfb7acf: link is not ready
[19712961.289084] IPVS: Creating netns size=2104 id=105
[19712961.356665] eth0: renamed from vethb91eb46
[19712961.388291] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[19712961.393660] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[19712961.400400] IPv6: ADDRCONF(NETDEV_CHANGE): vethdfb7acf: link becomes ready
[19712961.407766] docker0: port 1(vethdfb7acf) entered blocking state
[19712961.413499] docker0: port 1(vethdfb7acf) entered forwarding state
[19712961.418615] IPv6: ADDRCONF(NETDEV_CHANGE): docker0: link becomes ready
[19712961.573943] docker0: port 1(vethdfb7acf) entered disabled state
[19712961.578961] vethb91eb46: renamed from eth0
[19712961.635787] docker0: port 1(vethdfb7acf) entered disabled state
[19712961.646022] device vethdfb7acf left promiscuous mode
[19712961.648531] docker0: port 1(vethdfb7acf) entered disabled state
[19712972.012333] docker0: port 1(veth3a2d1d9) entered blocking state
[19712972.017902] docker0: port 1(veth3a2d1d9) entered disabled state
[19712972.025575] device veth3a2d1d9 entered promiscuous mode
[19712972.030615] IPv6: ADDRCONF(NETDEV_UP): veth3a2d1d9: link is not ready
[19712972.095017] IPVS: Creating netns size=2104 id=106
[19712972.164433] eth0: renamed from vethfdec5a6
[19712972.188405] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[19712972.192584] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[19712972.195963] IPv6: ADDRCONF(NETDEV_CHANGE): veth3a2d1d9: link becomes ready
[19712972.199901] docker0: port 1(veth3a2d1d9) entered blocking state
[19712972.203164] docker0: port 1(veth3a2d1d9) entered forwarding state
[19713091.805140] docker0: port 1(veth3a2d1d9) entered disabled state
[19713091.810114] vethfdec5a6: renamed from eth0
[19713091.860572] docker0: port 1(veth3a2d1d9) entered disabled state
[19713091.868719] device veth3a2d1d9 left promiscuous mode
[19713091.873245] docker0: port 1(veth3a2d1d9) entered disabled state
[19713966.748920] IPVS: Creating netns size=2104 id=107
[19713966.944037] eth0: renamed from veth33569eec
[19713966.978031] IPv6: ADDRCONF(NETDEV_UP): veth0: link is not ready
[19713967.000554] IPv6: eth0: IPv6 duplicate address fe80::8b5:1fff:fe03:d1d6 detected!
[19713967.000790] IPv6: ADDRCONF(NETDEV_UP): veth433971d5: link is not ready
[19713967.000797] IPv6: ADDRCONF(NETDEV_CHANGE): veth433971d5: link becomes ready
[19713967.000890] IPv6: ADDRCONF(NETDEV_CHANGE): veth0: link becomes ready
[19715406.152861] Bluetooth: Core ver 2.22
[19715406.156479] NET: Registered protocol family 31
[19715406.160376] Bluetooth: HCI device and connection manager initialized
[19715406.166372] Bluetooth: HCI socket layer initialized
[19715406.169930] Bluetooth: L2CAP socket layer initialized
[19715406.174984] Bluetooth: SCO socket layer initialized

JiriHorky commented 5 years ago

Thank you @dbyron0 for stepping in and providing the output!

One more think I wanted to point out @PaulFurtado, that I probably didn't stress out enough - the strange thing is that the new pod "jhorky..." can not ping the old pod, whereas it works the other way! And what's strange is that when the ping works , I still didn't see any ICMP packets in the tcpdump running on the affected "jhorky..." pod (nor the compute node) so I have no idea wtf was actually replying to those pings...

PaulFurtado commented 5 years ago

Summary of useful stuff I pulled from all those things pasted:

fluentd-loggly-hb9lf:
- ip: 100.96.236.178
- veth mac: ??? (missing ip addr output from inside pod, but host's ARP cache shows d2:5d:eb:14:bf:f1)
- host veth: veth3bb00fe4 (link-netnsid: 0, mac: 8e:d2:b6:7e:fa:06)
jhorky-shell
- ip: 100.96.254.43
- veth mac: 0a:70:58:d2:9d:d6
- host veth: veth433971d5 (link-netnsid: 18, mac: 6a:0d:54:97:81:ec)
fluentd (100.96.236.178) can ping jhorky (100.96.254.43) but not the opposite way
both IPs are assigned to the eth1 ENI which has the MAC address: 0a:b5:1f:03:d1:d6

relevant jhorky-shell ipv4 routes:

default via 10.110.174.111 dev veth0 
10.110.172.0/22 via 100.96.224.1 dev eth0 
10.110.174.111 dev veth0 scope link 
100.96.0.0/16 via 100.96.224.1 dev eth0 
100.96.224.0/19 dev eth0 proto kernel scope link src 100.96.254.43 
100.97.0.0/19 via 100.96.224.1 dev eth0

relevant fluentd-loggly-hb9lf ipv4 routes:

default via 10.110.174.111 dev veth0 
10.110.172.0/22 via 100.96.224.1 dev eth0 
10.110.174.111 dev veth0 scope link 
100.96.0.0/16 via 100.96.224.1 dev eth0 
100.96.224.0/19 dev eth0 proto kernel scope link src 100.96.236.178 
100.97.0.0/19 via 100.96.224.1 dev eth0

So jhorky-shell (100.96.254.43) should be dumping the packets destined for 100.96.236.178 directly into eth0 and ipvlan should figure out where they go internally.

ARP stuff:

Note that all ipvlan slave interfaces share the same MAC address as the ENI so the ARP responses for both IPs should return 0a:b5:1f:03:d1:d6 as the MAC. However, when the host hits the pods, it should send packets to the MAC address of the veth adapter.

For the successful ping from fluentd to jhorky, tcpdump inside jhorky shows the correct ARP reply being sent:

15:57:22.065934 ARP, Request who-has 100.96.254.43 tell 100.96.236.178, length 28
15:57:22.066041 ARP, Reply 100.96.254.43 is-at 0a:b5:1f:03:d1:d6, length 28

However, the compute node shows an odd ARP reply:

15:57:16.241711 ARP, Request who-has 100.96.236.178 tell 10.110.174.111, length 28
15:57:16.241732 ARP, Reply 100.96.236.178 is-at d2:5d:eb:14:bf:f1, length 28

In this case, the host's eth0 IP (10.110.174.111) wants to know where 100.96.236.178 (fluentd) is and the response is d2:5d:eb:14:bf:f1 which is theoretically the veth MAC for fluentd. The question is: why would the host be doing an ARP lookup for the pod at that time? I'm guessing kubelet is executing a health check against the fluentd pod maybe? If so this is simply unrelated. If not, something weird is happening (like arp proxy being turned on)

Also odd: jhorky's ARP cache is 100% empty. But maybe that's just because it hadn't pinged anything recently.

More stuff I'm interested in:

sysctl -a | grep net.ipv4.*eth on the host, and in both pods (can use nsenter --net for the pods)
ip addr in the fluentd pod
Output of iptables-save in both of the pods (likely need nsenter --net for this one). This will tell us if maybe iptables rules are doing something weird.
Can the host ping both pod IPs successfully? Can both pod IPs ping the host IP (10.110.174.111)?
Whenever you run tcpdump, use tcpdump -nnep -i any, this will perform no reverse DNS lookups and will show the source/dest mac addresses.
- Can you run this in parallel for both pods and the host at the same time and attempt the pings in either direction?
- Same, but ping the EC2 subnet gateway: 100.96.224.1
lsns --type=net on the host (maybe a sneaky stale namespace still has 100.96.254.43 claimed?)
- Also, run this nasty xargs command as root (switch to root instead of using sudo):
```
lsns --type=net --output=pid --noheadings | xargs -I{} sh -c 'echo "ip addr output for pid {} (hostname: $(nsenter -t {} --uts hostname)) namespace:" && nsenter --net -t {} ip addr && echo -------------'
```
  This will give us the IP addresses and hostnames of every namespace on the host to see if any stale namespace is hogging 100.96.254.43

splichy commented 5 years ago

Hi, I work together /w @JiriHorky @dbyron0

sysctl -a | grep net.ipv4.*eth on host

net.ipv4.conf.eth0.accept_local = 0
net.ipv4.conf.eth0.accept_redirects = 1
net.ipv4.conf.eth0.accept_source_route = 1
net.ipv4.conf.eth0.arp_accept = 0
net.ipv4.conf.eth0.arp_announce = 0
net.ipv4.conf.eth0.arp_filter = 0
net.ipv4.conf.eth0.arp_ignore = 0
net.ipv4.conf.eth0.arp_notify = 0
net.ipv4.conf.eth0.bootp_relay = 0
net.ipv4.conf.eth0.disable_policy = 0
net.ipv4.conf.eth0.disable_xfrm = 0
net.ipv4.conf.eth0.drop_gratuitous_arp = 0
net.ipv4.conf.eth0.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.eth0.force_igmp_version = 0
net.ipv4.conf.eth0.forwarding = 1
net.ipv4.conf.eth0.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.eth0.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.eth0.ignore_routes_with_linkdown = 0
net.ipv4.conf.eth0.log_martians = 0
net.ipv4.conf.eth0.mc_forwarding = 0
net.ipv4.conf.eth0.medium_id = 0
net.ipv4.conf.eth0.promote_secondaries = 0
net.ipv4.conf.eth0.proxy_arp = 0
net.ipv4.conf.eth0.proxy_arp_pvlan = 0
net.ipv4.conf.eth0.route_localnet = 0
net.ipv4.conf.eth0.rp_filter = 2
net.ipv4.conf.eth0.secure_redirects = 1
net.ipv4.conf.eth0.send_redirects = 1
net.ipv4.conf.eth0.shared_media = 1
net.ipv4.conf.eth0.src_valid_mark = 0
net.ipv4.conf.eth0.tag = 0
net.ipv4.conf.eth1.accept_local = 0
net.ipv4.conf.eth1.accept_redirects = 1
net.ipv4.conf.eth1.accept_source_route = 1
net.ipv4.conf.eth1.arp_accept = 0
net.ipv4.conf.eth1.arp_announce = 0
net.ipv4.conf.eth1.arp_filter = 0
net.ipv4.conf.eth1.arp_ignore = 0
net.ipv4.conf.eth1.arp_notify = 0
net.ipv4.conf.eth1.bootp_relay = 0
net.ipv4.conf.eth1.disable_policy = 0
net.ipv4.conf.eth1.disable_xfrm = 0
net.ipv4.conf.eth1.drop_gratuitous_arp = 0
net.ipv4.conf.eth1.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.eth1.force_igmp_version = 0
net.ipv4.conf.eth1.forwarding = 1
net.ipv4.conf.eth1.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.eth1.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.eth1.ignore_routes_with_linkdown = 0
net.ipv4.conf.eth1.log_martians = 0
net.ipv4.conf.eth1.mc_forwarding = 0
net.ipv4.conf.eth1.medium_id = 0
net.ipv4.conf.eth1.promote_secondaries = 0
net.ipv4.conf.eth1.proxy_arp = 0
net.ipv4.conf.eth1.proxy_arp_pvlan = 0
net.ipv4.conf.eth1.route_localnet = 0
net.ipv4.conf.eth1.rp_filter = 0
net.ipv4.conf.eth1.secure_redirects = 1
net.ipv4.conf.eth1.send_redirects = 1
net.ipv4.conf.eth1.shared_media = 1
net.ipv4.conf.eth1.src_valid_mark = 0
net.ipv4.conf.eth1.tag = 0
net.ipv4.conf.eth2.accept_local = 0
net.ipv4.conf.eth2.accept_redirects = 1
net.ipv4.conf.eth2.accept_source_route = 1
net.ipv4.conf.eth2.arp_accept = 0
net.ipv4.conf.eth2.arp_announce = 0
net.ipv4.conf.eth2.arp_filter = 0
net.ipv4.conf.eth2.arp_ignore = 0
net.ipv4.conf.eth2.arp_notify = 0
net.ipv4.conf.eth2.bootp_relay = 0
net.ipv4.conf.eth2.disable_policy = 0
net.ipv4.conf.eth2.disable_xfrm = 0
net.ipv4.conf.eth2.drop_gratuitous_arp = 0
net.ipv4.conf.eth2.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.eth2.force_igmp_version = 0
net.ipv4.conf.eth2.forwarding = 1
net.ipv4.conf.eth2.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.eth2.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.eth2.ignore_routes_with_linkdown = 0
net.ipv4.conf.eth2.log_martians = 0
net.ipv4.conf.eth2.mc_forwarding = 0
net.ipv4.conf.eth2.medium_id = 0
net.ipv4.conf.eth2.promote_secondaries = 0
net.ipv4.conf.eth2.proxy_arp = 0
net.ipv4.conf.eth2.proxy_arp_pvlan = 0
net.ipv4.conf.eth2.route_localnet = 0
net.ipv4.conf.eth2.rp_filter = 0
net.ipv4.conf.eth2.secure_redirects = 1
net.ipv4.conf.eth2.send_redirects = 1
net.ipv4.conf.eth2.shared_media = 1
net.ipv4.conf.eth2.src_valid_mark = 0
net.ipv4.conf.eth2.tag = 0
net.ipv4.conf.veth3bb00fe4.accept_local = 0
net.ipv4.conf.veth3bb00fe4.accept_redirects = 1
net.ipv4.conf.veth3bb00fe4.accept_source_route = 1
net.ipv4.conf.veth3bb00fe4.arp_accept = 0
net.ipv4.conf.veth3bb00fe4.arp_announce = 0
net.ipv4.conf.veth3bb00fe4.arp_filter = 0
net.ipv4.conf.veth3bb00fe4.arp_ignore = 0
net.ipv4.conf.veth3bb00fe4.arp_notify = 0
net.ipv4.conf.veth3bb00fe4.bootp_relay = 0
net.ipv4.conf.veth3bb00fe4.disable_policy = 0
net.ipv4.conf.veth3bb00fe4.disable_xfrm = 0
net.ipv4.conf.veth3bb00fe4.drop_gratuitous_arp = 0
net.ipv4.conf.veth3bb00fe4.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.veth3bb00fe4.force_igmp_version = 0
net.ipv4.conf.veth3bb00fe4.forwarding = 1
net.ipv4.conf.veth3bb00fe4.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.veth3bb00fe4.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.veth3bb00fe4.ignore_routes_with_linkdown = 0
net.ipv4.conf.veth3bb00fe4.log_martians = 0
net.ipv4.conf.veth3bb00fe4.mc_forwarding = 0
net.ipv4.conf.veth3bb00fe4.medium_id = 0
net.ipv4.conf.veth3bb00fe4.promote_secondaries = 0
net.ipv4.conf.veth3bb00fe4.proxy_arp = 0
net.ipv4.conf.veth3bb00fe4.proxy_arp_pvlan = 0
net.ipv4.conf.veth3bb00fe4.route_localnet = 0
net.ipv4.conf.veth3bb00fe4.rp_filter = 0
net.ipv4.conf.veth3bb00fe4.secure_redirects = 1
net.ipv4.conf.veth3bb00fe4.send_redirects = 1
net.ipv4.conf.veth3bb00fe4.shared_media = 1
net.ipv4.conf.veth3bb00fe4.src_valid_mark = 0
net.ipv4.conf.veth3bb00fe4.tag = 0
net.ipv4.conf.veth433971d5.accept_local = 0
net.ipv4.conf.veth433971d5.accept_redirects = 1
net.ipv4.conf.veth433971d5.accept_source_route = 1
net.ipv4.conf.veth433971d5.arp_accept = 0
net.ipv4.conf.veth433971d5.arp_announce = 0
net.ipv4.conf.veth433971d5.arp_filter = 0
net.ipv4.conf.veth433971d5.arp_ignore = 0
net.ipv4.conf.veth433971d5.arp_notify = 0
net.ipv4.conf.veth433971d5.bootp_relay = 0
net.ipv4.conf.veth433971d5.disable_policy = 0
net.ipv4.conf.veth433971d5.disable_xfrm = 0
net.ipv4.conf.veth433971d5.drop_gratuitous_arp = 0
net.ipv4.conf.veth433971d5.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.veth433971d5.force_igmp_version = 0
net.ipv4.conf.veth433971d5.forwarding = 1
net.ipv4.conf.veth433971d5.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.veth433971d5.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.veth433971d5.ignore_routes_with_linkdown = 0
net.ipv4.conf.veth433971d5.log_martians = 0
net.ipv4.conf.veth433971d5.mc_forwarding = 0
net.ipv4.conf.veth433971d5.medium_id = 0
net.ipv4.conf.veth433971d5.promote_secondaries = 0
net.ipv4.conf.veth433971d5.proxy_arp = 0
net.ipv4.conf.veth433971d5.proxy_arp_pvlan = 0
net.ipv4.conf.veth433971d5.route_localnet = 0
net.ipv4.conf.veth433971d5.rp_filter = 0
net.ipv4.conf.veth433971d5.secure_redirects = 1
net.ipv4.conf.veth433971d5.send_redirects = 1
net.ipv4.conf.veth433971d5.shared_media = 1
net.ipv4.conf.veth433971d5.src_valid_mark = 0
net.ipv4.conf.veth433971d5.tag = 0
net.ipv4.conf.vethd6d10032.accept_local = 0
net.ipv4.conf.vethd6d10032.accept_redirects = 1
net.ipv4.conf.vethd6d10032.accept_source_route = 1
net.ipv4.conf.vethd6d10032.arp_accept = 0
net.ipv4.conf.vethd6d10032.arp_announce = 0
net.ipv4.conf.vethd6d10032.arp_filter = 0
net.ipv4.conf.vethd6d10032.arp_ignore = 0
net.ipv4.conf.vethd6d10032.arp_notify = 0
net.ipv4.conf.vethd6d10032.bootp_relay = 0
net.ipv4.conf.vethd6d10032.disable_policy = 0
net.ipv4.conf.vethd6d10032.disable_xfrm = 0
net.ipv4.conf.vethd6d10032.drop_gratuitous_arp = 0
net.ipv4.conf.vethd6d10032.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.vethd6d10032.force_igmp_version = 0
net.ipv4.conf.vethd6d10032.forwarding = 1
net.ipv4.conf.vethd6d10032.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.vethd6d10032.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.vethd6d10032.ignore_routes_with_linkdown = 0
net.ipv4.conf.vethd6d10032.log_martians = 0
net.ipv4.conf.vethd6d10032.mc_forwarding = 0
net.ipv4.conf.vethd6d10032.medium_id = 0
net.ipv4.conf.vethd6d10032.promote_secondaries = 0
net.ipv4.conf.vethd6d10032.proxy_arp = 0
net.ipv4.conf.vethd6d10032.proxy_arp_pvlan = 0
net.ipv4.conf.vethd6d10032.route_localnet = 0
net.ipv4.conf.vethd6d10032.rp_filter = 0
net.ipv4.conf.vethd6d10032.secure_redirects = 1
net.ipv4.conf.vethd6d10032.send_redirects = 1
net.ipv4.conf.vethd6d10032.shared_media = 1
net.ipv4.conf.vethd6d10032.src_valid_mark = 0
net.ipv4.conf.vethd6d10032.tag = 0
net.ipv4.neigh.eth0.anycast_delay = 100
net.ipv4.neigh.eth0.app_solicit = 0
net.ipv4.neigh.eth0.base_reachable_time_ms = 30000
net.ipv4.neigh.eth0.delay_first_probe_time = 5
net.ipv4.neigh.eth0.gc_stale_time = 60
net.ipv4.neigh.eth0.locktime = 100
net.ipv4.neigh.eth0.mcast_resolicit = 0
net.ipv4.neigh.eth0.mcast_solicit = 3
net.ipv4.neigh.eth0.proxy_delay = 80
net.ipv4.neigh.eth0.proxy_qlen = 64
net.ipv4.neigh.eth0.retrans_time_ms = 1000
net.ipv4.neigh.eth0.ucast_solicit = 3
net.ipv4.neigh.eth0.unres_qlen = 31
net.ipv4.neigh.eth0.unres_qlen_bytes = 65536
net.ipv4.neigh.eth1.anycast_delay = 100
net.ipv4.neigh.eth1.app_solicit = 0
net.ipv4.neigh.eth1.base_reachable_time_ms = 30000
net.ipv4.neigh.eth1.delay_first_probe_time = 5
net.ipv4.neigh.eth1.gc_stale_time = 60
net.ipv4.neigh.eth1.locktime = 100
net.ipv4.neigh.eth1.mcast_resolicit = 0
net.ipv4.neigh.eth1.mcast_solicit = 3
net.ipv4.neigh.eth1.proxy_delay = 80
net.ipv4.neigh.eth1.proxy_qlen = 64
net.ipv4.neigh.eth1.retrans_time_ms = 1000
net.ipv4.neigh.eth1.ucast_solicit = 3
net.ipv4.neigh.eth1.unres_qlen = 31
net.ipv4.neigh.eth1.unres_qlen_bytes = 65536
net.ipv4.neigh.eth2.anycast_delay = 100
net.ipv4.neigh.eth2.app_solicit = 0
net.ipv4.neigh.eth2.base_reachable_time_ms = 30000
net.ipv4.neigh.eth2.delay_first_probe_time = 5
net.ipv4.neigh.eth2.gc_stale_time = 60
net.ipv4.neigh.eth2.locktime = 100
net.ipv4.neigh.eth2.mcast_resolicit = 0
net.ipv4.neigh.eth2.mcast_solicit = 3
net.ipv4.neigh.eth2.proxy_delay = 80
net.ipv4.neigh.eth2.proxy_qlen = 64
net.ipv4.neigh.eth2.retrans_time_ms = 1000
net.ipv4.neigh.eth2.ucast_solicit = 3
net.ipv4.neigh.eth2.unres_qlen = 31
net.ipv4.neigh.eth2.unres_qlen_bytes = 65536
net.ipv4.neigh.veth3bb00fe4.anycast_delay = 100
net.ipv4.neigh.veth3bb00fe4.app_solicit = 0
net.ipv4.neigh.veth3bb00fe4.base_reachable_time_ms = 30000
net.ipv4.neigh.veth3bb00fe4.delay_first_probe_time = 5
net.ipv4.neigh.veth3bb00fe4.gc_stale_time = 60
net.ipv4.neigh.veth3bb00fe4.locktime = 100
net.ipv4.neigh.veth3bb00fe4.mcast_resolicit = 0
net.ipv4.neigh.veth3bb00fe4.mcast_solicit = 3
net.ipv4.neigh.veth3bb00fe4.proxy_delay = 80
net.ipv4.neigh.veth3bb00fe4.proxy_qlen = 64
net.ipv4.neigh.veth3bb00fe4.retrans_time_ms = 1000
net.ipv4.neigh.veth3bb00fe4.ucast_solicit = 3
net.ipv4.neigh.veth3bb00fe4.unres_qlen = 31
net.ipv4.neigh.veth3bb00fe4.unres_qlen_bytes = 65536
net.ipv4.neigh.veth433971d5.anycast_delay = 100
net.ipv4.neigh.veth433971d5.app_solicit = 0
net.ipv4.neigh.veth433971d5.base_reachable_time_ms = 30000
net.ipv4.neigh.veth433971d5.delay_first_probe_time = 5
net.ipv4.neigh.veth433971d5.gc_stale_time = 60
net.ipv4.neigh.veth433971d5.locktime = 100
net.ipv4.neigh.veth433971d5.mcast_resolicit = 0
net.ipv4.neigh.veth433971d5.mcast_solicit = 3
net.ipv4.neigh.veth433971d5.proxy_delay = 80
net.ipv4.neigh.veth433971d5.proxy_qlen = 64
net.ipv4.neigh.veth433971d5.retrans_time_ms = 1000
net.ipv4.neigh.veth433971d5.ucast_solicit = 3
net.ipv4.neigh.veth433971d5.unres_qlen = 31
net.ipv4.neigh.veth433971d5.unres_qlen_bytes = 65536
net.ipv4.neigh.vethd6d10032.anycast_delay = 100
net.ipv4.neigh.vethd6d10032.app_solicit = 0
net.ipv4.neigh.vethd6d10032.base_reachable_time_ms = 30000
net.ipv4.neigh.vethd6d10032.delay_first_probe_time = 5
net.ipv4.neigh.vethd6d10032.gc_stale_time = 60
net.ipv4.neigh.vethd6d10032.locktime = 100
net.ipv4.neigh.vethd6d10032.mcast_resolicit = 0
net.ipv4.neigh.vethd6d10032.mcast_solicit = 3
net.ipv4.neigh.vethd6d10032.proxy_delay = 80
net.ipv4.neigh.vethd6d10032.proxy_qlen = 64
net.ipv4.neigh.vethd6d10032.retrans_time_ms = 1000
net.ipv4.neigh.vethd6d10032.ucast_solicit = 3
net.ipv4.neigh.vethd6d10032.unres_qlen = 31
net.ipv4.neigh.vethd6d10032.unres_qlen_bytes = 65536

nsenter -t ${JHORKY_PID} -n sysctl -a | grep net.ipv4.*eth for both pods are same:

net.ipv4.conf.eth0.accept_local = 0
net.ipv4.conf.eth0.accept_redirects = 1
net.ipv4.conf.eth0.accept_source_route = 1
net.ipv4.conf.eth0.arp_accept = 0
net.ipv4.conf.eth0.arp_announce = 0
net.ipv4.conf.eth0.arp_filter = 0
net.ipv4.conf.eth0.arp_ignore = 0
net.ipv4.conf.eth0.arp_notify = 0
net.ipv4.conf.eth0.bootp_relay = 0
net.ipv4.conf.eth0.disable_policy = 0
net.ipv4.conf.eth0.disable_xfrm = 0
net.ipv4.conf.eth0.drop_gratuitous_arp = 0
net.ipv4.conf.eth0.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.eth0.force_igmp_version = 0
net.ipv4.conf.eth0.forwarding = 1
net.ipv4.conf.eth0.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.eth0.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.eth0.ignore_routes_with_linkdown = 0
net.ipv4.conf.eth0.log_martians = 0
net.ipv4.conf.eth0.mc_forwarding = 0
net.ipv4.conf.eth0.medium_id = 0
net.ipv4.conf.eth0.promote_secondaries = 0
net.ipv4.conf.eth0.proxy_arp = 0
net.ipv4.conf.eth0.proxy_arp_pvlan = 0
net.ipv4.conf.eth0.route_localnet = 0
net.ipv4.conf.eth0.rp_filter = 0
net.ipv4.conf.eth0.secure_redirects = 1
net.ipv4.conf.eth0.send_redirects = 1
net.ipv4.conf.eth0.shared_media = 1
net.ipv4.conf.eth0.src_valid_mark = 0
net.ipv4.conf.eth0.tag = 0
net.ipv4.conf.veth0.accept_local = 0
net.ipv4.conf.veth0.accept_redirects = 1
net.ipv4.conf.veth0.accept_source_route = 1
net.ipv4.conf.veth0.arp_accept = 0
net.ipv4.conf.veth0.arp_announce = 0
net.ipv4.conf.veth0.arp_filter = 0
net.ipv4.conf.veth0.arp_ignore = 0
net.ipv4.conf.veth0.arp_notify = 0
net.ipv4.conf.veth0.bootp_relay = 0
net.ipv4.conf.veth0.disable_policy = 0
net.ipv4.conf.veth0.disable_xfrm = 0
net.ipv4.conf.veth0.drop_gratuitous_arp = 0
net.ipv4.conf.veth0.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.veth0.force_igmp_version = 0
net.ipv4.conf.veth0.forwarding = 1
net.ipv4.conf.veth0.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.veth0.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.veth0.ignore_routes_with_linkdown = 0
net.ipv4.conf.veth0.log_martians = 0
net.ipv4.conf.veth0.mc_forwarding = 0
net.ipv4.conf.veth0.medium_id = 0
net.ipv4.conf.veth0.promote_secondaries = 0
net.ipv4.conf.veth0.proxy_arp = 0
net.ipv4.conf.veth0.proxy_arp_pvlan = 0
net.ipv4.conf.veth0.route_localnet = 0
net.ipv4.conf.veth0.rp_filter = 0
net.ipv4.conf.veth0.secure_redirects = 1
net.ipv4.conf.veth0.send_redirects = 1
net.ipv4.conf.veth0.shared_media = 1
net.ipv4.conf.veth0.src_valid_mark = 0
net.ipv4.conf.veth0.tag = 0
net.ipv4.neigh.eth0.anycast_delay = 100
net.ipv4.neigh.eth0.app_solicit = 0
net.ipv4.neigh.eth0.base_reachable_time_ms = 30000
net.ipv4.neigh.eth0.delay_first_probe_time = 5
net.ipv4.neigh.eth0.gc_stale_time = 60
net.ipv4.neigh.eth0.locktime = 100
net.ipv4.neigh.eth0.mcast_resolicit = 0
net.ipv4.neigh.eth0.mcast_solicit = 3
net.ipv4.neigh.eth0.proxy_delay = 80
net.ipv4.neigh.eth0.proxy_qlen = 64
net.ipv4.neigh.eth0.retrans_time_ms = 1000
net.ipv4.neigh.eth0.ucast_solicit = 3
net.ipv4.neigh.eth0.unres_qlen = 31
net.ipv4.neigh.eth0.unres_qlen_bytes = 65536
net.ipv4.neigh.veth0.anycast_delay = 100
net.ipv4.neigh.veth0.app_solicit = 0
net.ipv4.neigh.veth0.base_reachable_time_ms = 30000
net.ipv4.neigh.veth0.delay_first_probe_time = 5
net.ipv4.neigh.veth0.gc_stale_time = 60
net.ipv4.neigh.veth0.locktime = 100
net.ipv4.neigh.veth0.mcast_resolicit = 0
net.ipv4.neigh.veth0.mcast_solicit = 3
net.ipv4.neigh.veth0.proxy_delay = 80
net.ipv4.neigh.veth0.proxy_qlen = 64
net.ipv4.neigh.veth0.retrans_time_ms = 1000
net.ipv4.neigh.veth0.ucast_solicit = 3
net.ipv4.neigh.veth0.unres_qlen = 31
net.ipv4.neigh.veth0.unres_qlen_bytes = 65536

nsenter -t ${FLUENTD_PID} -n ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
   valid_lft forever preferred_lft forever
inet6 ::1/128 scope host 
   valid_lft forever preferred_lft forever
2: eth0@veth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc noqueue state UNKNOWN group default 
link/ether 0a:b5:1f:03:d1:d6 brd ff:ff:ff:ff:ff:ff
inet 100.96.236.178/19 brd 100.96.255.255 scope global eth0
   valid_lft forever preferred_lft forever
inet6 fe80::8b5:1fff:fe03:d1d6/64 scope link 
   valid_lft forever preferred_lft forever
4: veth0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
link/ether d2:5d:eb:14:bf:f1 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::d05d:ebff:fe14:bff1/64 scope link 
   valid_lft forever preferred_lft forever

nsenter -t ${JHORKY_PID} -n iptables-save

# Generated by iptables-save v1.6.0 on Thu May 30 09:58:04 2019
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [9:756]
COMMIT
# Completed on Thu May 30 09:58:04 2019
# Generated by iptables-save v1.6.0 on Thu May 30 09:58:04 2019
*nat
:PREROUTING ACCEPT [2:170]
:INPUT ACCEPT [1:84]
:OUTPUT ACCEPT [10:842]
:POSTROUTING ACCEPT [2:170]
-A POSTROUTING -o eth0 -m comment --comment "kube-proxy SNAT" -j MASQUERADE
COMMIT
# Completed on Thu May 30 09:58:04 2019

nsenter -t ${FLUENTD_PID} -n iptables-save

# Generated by iptables-save v1.6.0 on Thu May 30 09:58:15 2019
*nat
:PREROUTING ACCEPT [36157920:2962177993]
:INPUT ACCEPT [8188:687792]
:OUTPUT ACCEPT [39778172:3179220003]
:POSTROUTING ACCEPT [39778163:3179219271]
-A POSTROUTING -o eth0 -m comment --comment "kube-proxy SNAT" -j MASQUERADE
COMMIT
# Completed on Thu May 30 09:58:15 2019

nsenter -t ${JHORKY_PID} -n ping 100.96.236.178 -c 2; nsenter -t ${JHORKY_PID} -n ping 100.96.224.1 -c 2; nsenter -t ${FLUENTD_PID} -n ping 100.96.254.43 -c 2; nsenter -t ${FLUENTD_PID} -n ping 100.96.224.1 -c 2
```
PING 100.96.236.178 (100.96.236.178) 56(84) bytes of data.
From 100.96.254.43 icmp_seq=1 Destination Host Unreachable
From 100.96.254.43 icmp_seq=2 Destination Host Unreachable
```

--- 100.96.236.178 ping statistics --- 2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1009ms pipe 2 PING 100.96.224.1 (100.96.224.1) 56(84) bytes of data. From 100.96.254.43 icmp_seq=1 Destination Host Unreachable From 100.96.254.43 icmp_seq=2 Destination Host Unreachable

--- 100.96.224.1 ping statistics --- 2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1021ms pipe 2 PING 100.96.254.43 (100.96.254.43) 56(84) bytes of data. 64 bytes from 100.96.254.43: icmp_seq=1 ttl=64 time=0.199 ms 64 bytes from 100.96.254.43: icmp_seq=2 ttl=64 time=0.069 ms

--- 100.96.254.43 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1021ms rtt min/avg/max/mdev = 0.069/0.134/0.199/0.065 ms PING 100.96.224.1 (100.96.224.1) 56(84) bytes of data. 64 bytes from 100.96.224.1: icmp_seq=1 ttl=255 time=0.265 ms 64 bytes from 100.96.224.1: icmp_seq=2 ttl=255 time=0.333 ms

--- 100.96.224.1 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1021ms rtt min/avg/max/mdev = 0.265/0.299/0.333/0.034 ms

`tcpdump -nnep -i any not tcp and not udp` on host

10:24:50.513843 Out 0a:b5:1f:03:d1:d6 ethertype ARP (0x0806), length 44: Request who-has 100.96.236.178 tell 100.96.254.43, length 28 10:24:51.537839 Out 0a:b5:1f:03:d1:d6 ethertype ARP (0x0806), length 44: Request who-has 100.96.236.178 tell 100.96.254.43, length 28 10:24:52.564815 Out 0a:b5:1f:03:d1:d6 ethertype ARP (0x0806), length 44: Request who-has 100.96.224.1 tell 100.96.254.43, length 28 10:24:52.564853 In 0a:bd:49:18:1d:5a ethertype ARP (0x0806), length 62: Reply 100.96.224.1 is-at 0a:bd:49:18:1d:5a, length 46 10:24:53.585834 Out 0a:b5:1f:03:d1:d6 ethertype ARP (0x0806), length 44: Request who-has 100.96.224.1 tell 100.96.254.43, length 28 10:24:53.585872 In 0a:bd:49:18:1d:5a ethertype ARP (0x0806), length 62: Reply 100.96.224.1 is-at 0a:bd:49:18:1d:5a, length 46 10:24:54.193770 Out 0a:b5:1f:03:d1:d6 ethertype ARP (0x0806), length 44: Request who-has 100.96.224.1 tell 100.96.248.69, length 28 10:24:54.193820 In 0a:bd:49:18:1d:5a ethertype ARP (0x0806), length 62: Reply 100.96.224.1 is-at 0a:bd:49:18:1d:5a, length 46 10:24:54.609828 Out 0a:b5:1f:03:d1:d6 ethertype ARP (0x0806), length 44: Request who-has 100.96.224.1 tell 100.96.254.43, length 28 10:24:54.609872 In 0a:bd:49:18:1d:5a ethertype ARP (0x0806), length 62: Reply 100.96.224.1 is-at 0a:bd:49:18:1d:5a, length 46 10:24:55.636481 Out 0a:b5:1f:03:d1:d6 ethertype ARP (0x0806), length 44: Request who-has 100.96.236.178 tell 100.96.254.43, length 28 10:24:56.660259 Out 0a:b5:1f:03:d1:d6 ethertype IPv4 (0x0800), length 100: 100.96.236.178 > 100.96.224.1: ICMP echo request, id 18769, seq 1, length 64 10:24:56.660453 In 0a:bd:49:18:1d:5a ethertype IPv4 (0x0800), length 100: 100.96.224.1 > 100.96.236.178: ICMP echo reply, id 18769, seq 1, length 64 10:24:57.681844 Out 0a:b5:1f:03:d1:d6 ethertype IPv4 (0x0800), length 100: 100.96.236.178 > 100.96.224.1: ICMP echo request, id 18769, seq 2, length 64 10:24:57.682002 In 0a:bd:49:18:1d:5a ethertype IPv4 (0x0800), length 100: 100.96.224.1 > 100.96.236.178: ICMP echo reply, id 18769, seq 2, length 64 ^C 15 packets captured 17 packets received by filter 0 packets dropped by kernel


`nsenter -t ${JHORKY_PID} -n tcpdump -nnep -i any`

10:24:50.513770 Out 0a:b5:1f:03:d1:d6 ethertype ARP (0x0806), length 44: Request who-has 100.96.236.178 tell 100.96.254.43, length 28 10:24:51.537769 Out 0a:b5:1f:03:d1:d6 ethertype ARP (0x0806), length 44: Request who-has 100.96.236.178 tell 100.96.254.43, length 28 10:24:52.561825 In 00:00:00:00:00:00 ethertype IPv4 (0x0800), length 128: 100.96.254.43 > 100.96.254.43: ICMP host 100.96.236.178 unreachable, length 92 10:24:52.561844 In 00:00:00:00:00:00 ethertype IPv4 (0x0800), length 128: 100.96.254.43 > 100.96.254.43: ICMP host 100.96.236.178 unreachable, length 92 10:24:52.564762 Out 0a:b5:1f:03:d1:d6 ethertype ARP (0x0806), length 44: Request who-has 100.96.224.1 tell 100.96.254.43, length 28 10:24:53.585772 Out 0a:b5:1f:03:d1:d6 ethertype ARP (0x0806), length 44: Request who-has 100.96.224.1 tell 100.96.254.43, length 28 10:24:54.609770 Out 0a:b5:1f:03:d1:d6 ethertype ARP (0x0806), length 44: Request who-has 100.96.224.1 tell 100.96.254.43, length 28 10:24:55.633819 In 00:00:00:00:00:00 ethertype IPv4 (0x0800), length 128: 100.96.254.43 > 100.96.254.43: ICMP host 100.96.224.1 unreachable, length 92 10:24:55.633828 In 00:00:00:00:00:00 ethertype IPv4 (0x0800), length 128: 100.96.254.43 > 100.96.254.43: ICMP host 100.96.224.1 unreachable, length 92 10:24:55.636478 B 0a:b5:1f:03:d1:d6 ethertype ARP (0x0806), length 44: Request who-has 100.96.236.178 tell 100.96.254.43, length 28 ^C 10 packets captured 14 packets received by filter 0 packets dropped by kernel


`nsenter -t ${FLUENTD_PID} -n tcpdump -nnep -i any`

10:24:50.513825 B 0a:b5:1f:03:d1:d6 ethertype ARP (0x0806), length 44: Request who-has 100.96.236.178 tell 100.96.254.43, length 28 10:24:50.513868 Out 0a:b5:1f:03:d1:d6 ethertype ARP (0x0806), length 44: Reply 100.96.236.178 is-at 0a:b5:1f:03:d1:d6, length 28 10:24:51.537821 B 0a:b5:1f:03:d1:d6 ethertype ARP (0x0806), length 44: Request who-has 100.96.236.178 tell 100.96.254.43, length 28 10:24:51.537864 Out 0a:b5:1f:03:d1:d6 ethertype ARP (0x0806), length 44: Reply 100.96.236.178 is-at 0a:b5:1f:03:d1:d6, length 28 10:24:52.564798 B 0a:b5:1f:03:d1:d6 ethertype ARP (0x0806), length 44: Request who-has 100.96.224.1 tell 100.96.254.43, length 28 10:24:53.585819 B 0a:b5:1f:03:d1:d6 ethertype ARP (0x0806), length 44: Request who-has 100.96.224.1 tell 100.96.254.43, length 28 10:24:54.609809 B 0a:b5:1f:03:d1:d6 ethertype ARP (0x0806), length 44: Request who-has 100.96.224.1 tell 100.96.254.43, length 28 10:24:55.636401 Out 0a:b5:1f:03:d1:d6 ethertype IPv4 (0x0800), length 100: 100.96.236.178 > 100.96.254.43: ICMP echo request, id 18768, seq 1, length 64 10:24:55.636463 B 0a:b5:1f:03:d1:d6 ethertype ARP (0x0806), length 44: Request who-has 100.96.236.178 tell 100.96.254.43, length 28 10:24:55.636504 Out 0a:b5:1f:03:d1:d6 ethertype ARP (0x0806), length 44: Reply 100.96.236.178 is-at 0a:b5:1f:03:d1:d6, length 28 10:24:55.636576 In 0a:b5:1f:03:d1:d6 ethertype IPv4 (0x0800), length 100: 100.96.254.43 > 100.96.236.178: ICMP echo reply, id 18768, seq 1, length 64 10:24:56.657811 Out 0a:b5:1f:03:d1:d6 ethertype IPv4 (0x0800), length 100: 100.96.236.178 > 100.96.254.43: ICMP echo request, id 18768, seq 2, length 64 10:24:56.657849 In 0a:b5:1f:03:d1:d6 ethertype IPv4 (0x0800), length 100: 100.96.254.43 > 100.96.236.178: ICMP echo reply, id 18768, seq 2, length 64 10:24:56.660243 Out 0a:b5:1f:03:d1:d6 ethertype IPv4 (0x0800), length 100: 100.96.236.178 > 100.96.224.1: ICMP echo request, id 18769, seq 1, length 64 10:24:56.660459 In 0a:bd:49:18:1d:5a ethertype IPv4 (0x0800), length 100: 100.96.224.1 > 100.96.236.178: ICMP echo reply, id 18769, seq 1, length 64 10:24:57.681828 Out 0a:b5:1f:03:d1:d6 ethertype IPv4 (0x0800), length 100: 100.96.236.178 > 100.96.224.1: ICMP echo request, id 18769, seq 2, length 64 10:24:57.682016 In 0a:bd:49:18:1d:5a ethertype IPv4 (0x0800), length 100: 100.96.224.1 > 100.96.236.178: ICMP echo reply, id 18769, seq 2, length 64 ^C 17 packets captured 17 packets received by filter 0 packets dropped by kernel



- `lsns --type=net` no sneaky stale namespace using this IP

BTW, right now an old(not evicted pods: fluentd, traefik) are able to comunicate with rest of cluster and also to the internet
But every new pod (jhorky-shell, vsplichal-shell) are not able to reach nowhere - but for both, something respons to ping also to pings from pods running on different nodes.

ngaya-ll commented 5 years ago

Also odd: jhorky's ARP cache is 100% empty. But maybe that's just because it hadn't pinged anything recently.

This is expected because it is not able to receive ARP responses for pod IPs. The pod is able to communicate with the node IP which shows up in the cache as expected.

/ # ping 10.110.174.111
PING 10.110.174.111 (10.110.174.111) 56(84) bytes of data.
64 bytes from 10.110.174.111: icmp_seq=1 ttl=64 time=0.087 ms
64 bytes from 10.110.174.111: icmp_seq=2 ttl=64 time=0.067 ms
^C
--- 10.110.174.111 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 7ms
rtt min/avg/max/mdev = 0.067/0.077/0.087/0.010 ms
/ # ip neigh
10.110.174.111 dev veth0 lladdr 6a:0d:54:97:81:ec REACHABLE

The affected pod is also able to ping public IPs

/ # ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=49 time=2.07 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=49 time=1.80 ms
^C
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 2ms
rtt min/avg/max/mdev = 1.798/1.935/2.072/0.137 ms

When attempting to ping another pod IP, that address briefly shows up in the cache as failed, then disappears after a moment.

/ # ping 100.96.236.178
PING 100.96.236.178 (100.96.236.178) 56(84) bytes of data.
From 100.96.254.43 icmp_seq=1 Destination Host Unreachable
From 100.96.254.43 icmp_seq=2 Destination Host Unreachable
From 100.96.254.43 icmp_seq=3 Destination Host Unreachable
^C
--- 100.96.236.178 ping statistics ---
5 packets transmitted, 0 received, +3 errors, 100% packet loss, time 95ms
pipe 4
/ # ip neigh
100.96.236.178 dev eth0  FAILED
10.110.174.111 dev veth0 lladdr 6a:0d:54:97:81:ec REACHABLE

lbernail commented 5 years ago

Out of curiosity, which version of kube-proxy are you using and in which mode?

An new ARP sysctl was introduced in 1.13 which broke ARP for us. Issue here: https://github.com/kubernetes/kubernetes/issues/71555

This only applies to a few versions of kube-proxy (it now defaults to the previous behavior) and only in IPVS mode (which you don't seem to be using based on the output of ip addr)

ngaya-ll commented 5 years ago

We are using k8s.gcr.io/kube-proxy:v1.10.7 in the default (iptables) mode, deployed with kops.

/usr/local/bin/kube-proxy
      --cluster-cidr=100.96.0.0/11 --conntrack-max-per-core=131072 --hostname-override=ip-10-110-174-111.eu-central-1.compute.internal
      --kubeconfig=/var/lib/kube-proxy/kubeconfig --master=https://api.internal.voda.kube.llabs.io
      --oom-score-adj=-998 --resource-container="" --v=2

lyft / cni-ipvlan-vpc-k8s

Pods loosing connection to each other on particular node from time to time #70

Summary of useful stuff I pulled from all those things pasted:

ARP stuff: