Closed slecrenski closed 6 years ago
slave:
tcpdump -n udp dst portrange 8472
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
19:59:45.951924 IP 10.244.0.100.33274 > 10.244.0.4.otv: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.244.0.100.sun-sr-https > 10.244.3.2.36460: Flags [S.], seq 3980964095, ack 1743480415, win 27960, options [mss 1410,sackOK,TS val 3518478 ecr 1223820,nop,wscale 7], length 0
19:59:45.952210 IP 10.244.0.100.58754 > 10.244.0.4.otv: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.244.0.100.sun-sr-https > 10.244.3.2.36462: Flags [S.], seq 696635755, ack 2548921951, win 27960, options [mss 1410,sackOK,TS val 3518479 ecr 1223820,nop,wscale 7], length 0
19:59:46.953154 IP 10.244.0.100.58754 > 10.244.0.4.otv: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.244.0.100.sun-sr-https > 10.244.3.2.36462: Flags [S.], seq 696635755, ack 2548921951, win 27960, options [mss 1410,sackOK,TS val 3519480 ecr 1223820,nop,wscale 7], length 0
19:59:46.954017 IP 10.244.0.100.33274 > 10.244.0.4.otv: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.244.0.100.sun-sr-https > 10.244.3.2.36460: Flags [S.], seq 3980964095, ack 1743480415, win 27960, options [mss 1410,sackOK,TS val 3519480 ecr 1223820,nop,wscale 7], length 0
19:59:46.954049 IP 10.244.0.100.58754 > 10.244.0.4.otv: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.244.0.100.sun-sr-https > 10.244.3.2.36462: Flags [S.], seq 696635755, ack 2548921951, win 27960, options [mss 1410,sackOK,TS val 3519480 ecr 1223820,nop,wscale 7], length 0
19:59:48.154188 IP 10.244.0.100.33274 > 10.244.0.4.otv: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.244.0.100.sun-sr-https > 10.244.3.2.36460: Flags [S.], seq 3980964095, ack 1743480415, win 27960, options [mss 1410,sackOK,TS val 3520681 ecr 1223820,nop,wscale 7], length 0
19:59:48.954142 IP 10.244.0.100.58754 > 10.244.0.4.otv: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.244.0.100.sun-sr-https > 10.244.3.2.36462: Flags [S.], seq 696635755, ack 2548921951, win 27960, options [mss 1410,sackOK,TS val 3521481 ecr 1223820,nop,wscale 7], length 0
19:59:48.955906 IP 10.244.0.100.33274 > 10.244.0.4.otv: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.244.0.100.sun-sr-https > 10.244.3.2.36460: Flags [S.], seq 3980964095, ack 1743480415, win 27960, options [mss 1410,sackOK,TS val 3521482 ecr 1223820,nop,wscale 7], length 0
19:59:48.955933 IP 10.244.0.100.58754 > 10.244.0.4.otv: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.244.0.100.sun-sr-https > 10.244.3.2.36462: Flags [S.], seq 696635755, ack 2548921951, win 27960, options [mss 1410,sackOK,TS val 3521482 ecr 1223820,nop,wscale 7], length 0
19:59:51.154189 IP 10.244.0.100.33274 > 10.244.0.4.otv: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.244.0.100.sun-sr-https > 10.244.3.2.36460: Flags [S.], seq 3980964095, ack 1743480415, win 27960, options [mss 1410,sackOK,TS val 3523681 ecr 1223820,nop,wscale 7], length 0
19:59:52.963897 IP 10.244.0.100.58754 > 10.244.0.4.otv: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.244.0.100.sun-sr-https > 10.244.3.2.36462: Flags [S.], seq 696635755, ack 2548921951, win 27960, options [mss 1410,sackOK,TS val 3525490 ecr 1223820,nop,wscale 7], length 0
master:
20:00:17.955855 IP 10.244.0.100.57275 > 10.244.0.4.otv: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.244.0.100.sun-sr-https > 10.244.3.2.36542: Flags [S.], seq 2016186502, ack 98983334, win 27960, options [mss 1410,sackOK,TS val 3550482 ecr 1254821,nop,wscale 7], length 0
20:00:19.155120 IP 10.244.0.100.46515 > 10.244.0.4.otv: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.244.0.100.sun-sr-https > 10.244.3.2.36540: Flags [S.], seq 3330915834, ack 368111801, win 27960, options [mss 1410,sackOK,TS val 3551682 ecr 1254820,nop,wscale 7], length 0
20:00:19.955223 IP 10.244.0.100.57275 > 10.244.0.4.otv: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.244.0.100.sun-sr-https > 10.244.3.2.36542: Flags [S.], seq 2016186502, ack 98983334, win 27960, options [mss 1410,sackOK,TS val 3552482 ecr 1254821,nop,wscale 7], length 0
20:00:19.955792 IP 10.244.0.100.46515 > 10.244.0.4.otv: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.244.0.100.sun-sr-https > 10.244.3.2.36540: Flags [S.], seq 3330915834, ack 368111801, win 27960, options [mss 1410,sackOK,TS val 3552482 ecr 1254820,nop,wscale 7], length 0
20:00:19.959782 IP 10.244.0.100.57275 > 10.244.0.4.otv: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.244.0.100.sun-sr-https > 10.244.3.2.36542: Flags [S.], seq 2016186502, ack 98983334, win 27960, options [mss 1410,sackOK,TS val 3552486 ecr 1254821,nop,wscale 7], length 0
20:00:22.155192 IP 10.244.0.100.46515 > 10.244.0.4.otv: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.244.0.100.sun-sr-https > 10.244.3.2.36540: Flags [S.], seq 3330915834, ack 368111801, win 27960, options [mss 1410,sackOK,TS val 3554682 ecr 1254820,nop,wscale 7], length 0
20:00:23.964762 IP 10.244.0.100.46515 > 10.244.0.4.otv: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.244.0.100.sun-sr-https > 10.244.3.2.36540: Flags [S.], seq 3330915834, ack 368111801, win 27960, options [mss 1410,sackOK,TS val 3556491 ecr 1254820,nop,wscale 7], length 0
20:00:23.971974 IP 10.244.0.100.57275 > 10.244.0.4.otv: OTV, flags [I] (0x08), overlay 0, instance 1
IP 10.244.0.100.sun-sr-https > 10.244.3.2.36542: Flags [S.], seq 2016186502, ack 98983334, win 27960, options [mss 1410,sackOK,TS val 3556498 ecr 1254821,nop,wscale 7], length 0
Figured it out.
Figured it out.
What have helped?
Docker: 1.12.6 RHEL: 7.3 Linux k8s-master 3.10.0-693.21.1.el7.x86_64 #1 SMP Fri Feb 23 18:54:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux Kubernetes 1.9.3 quay.io/calico/node:v2.6.2 quay.io/calico/cni:v1.11.0 quay.io/coreos/flannel:v0.9.1
Azure Cloud with vnet address space: 10.244.0.0/16
Cluster was initialized with
Cluster is running in Azure with the same virtual network as the pod cidr.
I have a very basic configuration. 1 Master Node and 1 Agent Node. DNS queries are not working on the agent node. the kube-dns is running on the master node. Master Node IP 10.244.0.100 and Agent Node IP: 10.244.0.4.
I am trying to figure out why it is that I cannot communicate with 10.96.0.10 (kube-dns) which is supposed to be routed to the master node (where kube-dns is running).
I've been looking at log files and enabling level 10 verbosity for the past several hours. What does this error message mean?
vxlan_network.go:158] failed to add vxlanRoute (10.244.0.0/24 -> 10.244.0.0): invalid argument
I am unable to get pods that require kube-dns to run. They just fail with a dns error trying to perform a lookup to kubernetes.default.svc.cluster.local. If i try to scale kube-dns to launch on the non-master node the kube-dns fails to start on that node due to an issue with dns lookup.
I am unable to get pod-to-kube-dns and node->kube-dns communication working. How can I debug what the issue is?
These are RHEL 7.3 nodes with:
Master Node:
Slave Node:
Interestingly enough I can do things like this
Slave Node Iptables:
Anyway to use tcpdump to figure out this issue?
What does this error mean? vxlan_network.go:158] failed to add vxlanRoute (10.244.0.0/24 -> 10.244.0.0): invalid argument
Master is running at 10.244.0.100 and agent node is running at 10.244.0.4.
--master
--agent