search

contiv / netplugin

Container networking for various use cases
Apache License 2.0
514 stars 177 forks source link

OpenFlow rules missing for service/node after certain period of depoloyment #1127

Open cloudrarch opened 6 years ago

cloudrarch commented 6 years ago

Description

I have deployed a customized k8s cluster and on top of that installed Contiv as CNI networking. All worked fine, but after 2 or 3 days noticed that OpenFlow rules for a specific service/node missing.

Expected Behavior

All the PODs should be accessible from any of the host systems. ie host ---> pod communication

Observed Behavior

Unable to connect to the POD and contivh1 interface IP from other hosts.Due to this unable to connect to the k8s services hosted on a specific host.

After some investigation, I found that the OpenFlow rules for that particular POD IP and contivh1 interface IP are missing.

The is sometime happening for POD IP and sometimes for contivh1 interface IP.

[root@]# kubectl get pod -n kube-system |grep contiv contiv-netmaster-bj2nc 1/1 Running 0 6d contiv-netmaster-kqkgq 1/1 Running 0 6d contiv-netplugin-6rrgc 1/1 Running 0 6d contiv-netplugin-fdbtd 1/1 Running 0 6d contiv-netplugin-j7qc9 1/1 Running 0 6d contiv-netplugin-k4p6q 1/1 Running 0 6d contiv-netplugin-kpnq9 1/1 Running 0 6d contiv-netplugin-p6k59 1/1 Running 0 6d contiv-netplugin-t6zbh 1/1 Running 0 6d contiv-netplugin-zsd9g 1/1 Running 0 6d [root@]#

[root@]# kubectl exec contiv-netplugin-6rrgc -n kube-system -- ovs-ofctl -O Openflow13 dump-flows contivVxlanBridge | grep 132.1.1.8 [root@s ~]#

[root@~]# kubectl exec contiv-netplugin-fdbtd -n kube-system -- ovs-ofctl -O Openflow13 dump-flows contivVxlanBridge | grep 132.1.1.8 cookie=0x1dd37, duration=9532.039s, table=4, n_packets=0, n_bytes=0, priority=100,ip,metadata=0x100000000/0xff00000000,nw_dst=132.1.1.8 actions=write_metadata:0/0xfffe,goto_table:5 cookie=0x1dd36, duration=9532.039s, table=7, n_packets=0, n_bytes=0, priority=100,ip,metadata=0x100000000/0xff00000000,nw_dst=132.1.1.8 actions=set_field:0x1->tun_id,set_field:00:00:11:11:11:11->eth_dst,output:267 [root@ ~]#

[root@ ~]# kubectl exec contiv-netplugin-j7qc9 -n kube-system -- ovs-ofctl -O Openflow13 dump-flows contivVxlanBridge | grep 132.1.1.8 cookie=0x499, duration=340671.602s, table=4, n_packets=86, n_bytes=8428, priority=100,ip,metadata=0x100000000/0xff00000000,nw_dst=132.1.1.8 actions=write_metadata:0/0xfffe,goto_table:5 cookie=0x498, duration=340671.603s, table=7, n_packets=86, n_bytes=8428, priority=100,ip,metadata=0x100000000/0xff00000000,nw_dst=132.1.1.8 actions=set_field:0x1->tun_id,set_field:00:00:11:11:11:11->eth_dst,output:12 [root@ ~]#

[root@ ~]# kubectl exec contiv-netplugin-k4p6q -n kube-system -- ovs-ofctl -O Openflow13 dump-flows contivVxlanBridge | grep 132.1.1.8 [root@ ~]#

[root@ ~]# kubectl exec contiv-netplugin-kpnq9 -n kube-system -- ovs-ofctl -O Openflow13 dump-flows contivVxlanBridge | grep 132.1.1.8 cookie=0x1f, duration=604562.586s, table=4, n_packets=329709, n_bytes=31594881, priority=100,ip,metadata=0x100000000/0xff00000000,nw_dst=132.1.1.8 actions=write_metadata:0/0xfffe,goto_table:5 cookie=0x1e, duration=604562.586s, table=7, n_packets=391455, n_bytes=49623858, priority=100,ip,metadata=0x100000000/0xff00000000,nw_dst=132.1.1.8 actions=set_field:00:00:11:11:11:11->eth_src,set_field:02:02:84:01:01:08->eth_dst,output:10 [root ~]#

[root@s~]# kubectl exec contiv-netplugin-p6k59 -n kube-system -- ovs-ofctl -O Openflow13 dump-flows contivVxlanBridge | grep 132.1.1.8 [root@~]#

[root@ ~]# kubectl exec contiv-netplugin-t6zbh -n kube-system -- ovs-ofctl -O Openflow13 dump-flows contivVxlanBridge | grep 132.1.1.8 cookie=0x49f, duration=340798.618s, table=4, n_packets=3, n_bytes=294, priority=100,ip,metadata=0x100000000/0xff00000000,nw_dst=132.1.1.8 actions=write_metadata:0/0xfffe,goto_table:5 cookie=0x49e, duration=340798.618s, table=7, n_packets=3, n_bytes=294, priority=100,ip,metadata=0x100000000/0xff00000000,nw_dst=132.1.1.8 actions=set_field:0x1->tun_id,set_field:00:00:11:11:11:11->eth_dst,output:13 [root@ ~]#

[root@ ~]# kubectl exec contiv-netplugin-zsd9g -n kube-system -- ovs-ofctl -O Openflow13 dump-flows contivVxlanBridge | grep 132.1.1.8 [root@ ~]#

Steps to Reproduce (for bugs)

Your Environment