HackToHell
commented
5 years ago Was able to isolate logs better, getting
WARN: 2019/02/28 16:09:51.536184 Vetoed installation of hairpin flow FlowSpec{keys: [InPortFlowKey{vport: 1} EthernetFlowKey{src: 12:cc:f5:35:61:28, dst: 66:2f:0f:7c:15:6d}], actions: [OutputAction{vport: 1}]}
WARN: 2019/02/28 16:09:52.536743 Vetoed installation of hairpin flow FlowSpec{keys: [EthernetFlowKey{src: 12:cc:f5:35:61:28, dst: 66:2f:0f:7c:15:6d} InPortFlowKey{vport: 1}], actions: [OutputAction{vport: 1}]}
WARN: 2019/02/28 16:09:54.540654 Vetoed installation of hairpin flow FlowSpec{keys: [EthernetFlowKey{src: 12:cc:f5:35:61:28, dst: 66:2f:0f:7c:15:6d} InPortFlowKey{vport: 1}], actions: [OutputAction{vport: 1}]}
WARN: 2019/02/28 16:09:58.548697 Vetoed installation of hairpin flow FlowSpec{keys: [EthernetFlowKey{src: 12:cc:f5:35:61:28, dst: 66:2f:0f:7c:15:6d} InPortFlowKey{vport: 1}], actions: [OutputAction{vport: 1}]}
WARN: 2019/02/28 16:10:06.570492 Vetoed installation of hairpin flow FlowSpec{keys: [EthernetFlowKey{src: 12:cc:f5:35:61:28, dst: 66:2f:0f:7c:15:6d} InPortFlowKey{vport: 1}], actions: [OutputAction{vport: 1}]}
WARN: 2019/02/28 16:10:08.568584 Vetoed installation of hairpin flow FlowSpec{keys: [EthernetFlowKey{src: 12:cc:f5:35:61:28, dst: 66:2f:0f:7c:15:6d} InPortFlowKey{vport: 1}], actions: [OutputAction{vport: 1}]}
IN12:cc:f5:35:61:28 is the weave bridge IP.
6: weave: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1376 qdisc noqueue state UP group default qlen 1000
link/ether 12:cc:f5:35:61:28 brd ff:ff:ff:ff:ff:ff
inet 10.32.200.0/16 brd 10.32.255.255 scope global weave
valid_lft forever preferred_lft forever
inet6 fe80::10cc:f5ff:fe35:6128/64 scope link
valid_lft forever preferred_lft forever
bboreham
What you expected to happen?
When a pod comes up it's able to hit any internal service consistently,
kubernetes.default.svcfor this issue.What happened?
Getting
error: couldn't get deployment devicebroker-12: Get https://172.30.0.1:443/api/v1/namespaces/v14-rapyuta-core/replicationcontrollers/devicebroker-12: dial tcp 172.30.0.1:443: i/o timeoutHow to reproduce it?
Happens with openshift deployer pods that bring up a new pods when a deployment is rolled, little sporadic but can be easily reproduced.
Anything else we need to know?
Looks like it was an issue in Openshift SDN and was fixed by changing a flow. https://github.com/openshift/origin/issues/5796 The fix (?) https://github.com/openshift/openshift-sdn/pull/236/files Since weave also uses ovs, could it be related?
Network policies are enabled for some name spaces in the cluster.
Versions:
Openshift 3.9 running on Microsoft Azure.
Openshift version
oc v3.9.0+ba7faec-1 kubernetes v1.9.1+a0ce1bc657 features: Basic-Auth GSSAPI Kerberos SPNEGO
Uname
Linux oc-master-0 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Weave
/home/weave # ./weave --local status
PeerDiscovery: enabled Targets: 9 Connections: 18 (17 established, 1 failed) Peers: 18 (with 306 established connections) TrustedSubnets: none
DefaultSubnet: 10.32.0.0/16
Docker
Client: Version: 1.13.1 API version: 1.26 Package version: docker-1.13.1-88.git07f3374.el7.centos.x86_64 Go version: go1.9.4 Git commit: 07f3374/1.13.1 Built: Fri Dec 7 16:13:51 2018 OS/Arch: linux/amd64
Server: Version: 1.13.1 API version: 1.26 (minimum version 1.12) Package version: docker-1.13.1-88.git07f3374.el7.centos.x86_64 Go version: go1.9.4 Git commit: 07f3374/1.13.1 Built: Fri Dec 7 16:13:51 2018 OS/Arch: linux/amd64 Experimental: false
Logs:
There's a lot of
Vetoed installation of hairpin flowmessages.Complete logs for weave at https://gist.github.com/HackToHell/5ffc79ca73f0bbf83c7697857fb34395
Startup logs for kubelet at https://gist.github.com/HackToHell/97605144a51a5850c8828ef5f45cb745
Network:
Routes
default via 10.2.0.1 dev eth0 proto dhcp metric 100 10.2.0.0/16 dev eth0 proto kernel scope link src 10.2.0.8 metric 100 10.32.0.0/16 dev weave proto kernel scope link src 10.32.16.0 168.63.129.16 via 10.2.0.1 dev eth0 proto dhcp metric 100 169.254.169.254 via 10.2.0.1 dev eth0 proto dhcp metric 100 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
Addrs
1: lo inet 127.0.0.1/8 scope host lo\ valid_lft forever preferred_lft forever 2: eth0 inet 10.2.0.8/16 brd 10.2.255.255 scope global noprefixroute eth0\ valid_lft forever preferred_lft forever 3: docker0 inet 172.17.0.1/16 scope global docker0\ valid_lft forever preferred_lft forever 6: weave inet 10.32.16.0/16 brd 10.32.255.255 scope global weave\ valid_lft forever preferred_lft forever
IP Tables Save
https://gist.github.com/HackToHell/73a249b0ab9818703905d976d41ab262
Output of
ip a