search

openvswitch / ovs-issues

Issue tracker repo for Open vSwitch
10 stars 3 forks source link

[Windows] Wrong invalid CT state #197

Open ruicao93 opened 3 years ago

ruicao93 commented 3 years ago

Description OVS treat normal reply as invalid CT state packet.

How to repro

  1. Prepare two Windows node and install OVS.
  2. Install one Pod/Container on each node and connect the Pod to OVS.
  3. Install OpenFlow rules on each node to make two Pods can talk with each other in NoEncap-mode. Following graph shows the traffic path on the node. topo

Traffic path:

route results: 192.168.193.0 255.255.255.0 10.176.25.211 10.176.27.178 271

  1. Test tcp connection and the connection will fail. We could see the reply pkts marked as invalid and are dropped.

    table=6, n_packets=40, n_bytes=15524, ct_state=+inv+trk,ip actions=drop

OpenFlow rules on node

 cookie=0x520, duration=56.688s, table=0, n_packets=0, n_bytes=0, priority=520,ip,in_port="Ethernet0 2",nw_src=192.168.193.0/24 actions=LOCAL 
 cookie=0x520, duration=61.504s, table=0, n_packets=3, n_bytes=198, priority=520,ip,in_port="powershe-0eefc6",nw_src=192.168.194.8,nw_dst=192.168.193.0/24 actions
=resubmit(,1) 
 cookie=0x520, duration=56.631s, table=0, n_packets=0, n_bytes=0, priority=520,ip,in_port="antrea-gw0",nw_src=192.168.193.0/24,nw_dst=192.168.194.8 actions=resubm
it(,1) 
 cookie=0x520, duration=61.053s, table=0, n_packets=0, n_bytes=0, priority=520,ip,in_port=LOCAL,nw_dst=192.168.193.0/24 actions=output:"Ethernet0 2"              

 cookie=0x520, duration=76.022s, table=1, n_packets=3, n_bytes=198, actions=drop 
 cookie=0x520, duration=70.383s, table=1, n_packets=0, n_bytes=0, priority=520,ip actions=ct(table=2,zone=64520,nat) 

 cookie=0x520, duration=2710.066s, table=2, n_packets=81, n_bytes=20431, priority=520,ip actions=resubmit(,3) 
 cookie=0x520, duration=2715.529s, table=2, n_packets=0, n_bytes=0, priority=0 actions=drop

 cookie=0x520, duration=2722.363s, table=3, n_packets=0, n_bytes=0, priority=0,ip actions=dec_ttl,resubmit(,4) 
 cookie=0x520, duration=2716.381s, table=3, n_packets=81, n_bytes=20431, ip actions=dec_ttl,resubmit(,4)                                                          
 cookie=0x520, duration=2716.808s, table=3, n_packets=0, n_bytes=0, priority=520,ip,nw_dst=192.168.193.0/24 actions=dec_ttl,mod_dl_dst:00:15:5d:1a:d4:0b,resubmit(
,4)

 cookie=0x520, duration=2725.493s, table=4, n_packets=34, n_bytes=3643, priority=520,dl_dst=00:15:5d:1a:d4:0b actions=load:0x2->NXM_NX_REG1[],load:0x1->NXM_NX_REG
0[16],resubmit(,6)                                                                                                                                                
 cookie=0x520, duration=2725.066s, table=4, n_packets=47, n_bytes=16788, priority=520,dl_dst=00:15:5d:61:83:41 actions=load:0x4->NXM_NX_REG1[],load:0x1->NXM_NX_RE
G0[16],resubmit(,6)                                                                                                                                               
 cookie=0x520, duration=2731.149s, table=4, n_packets=0, n_bytes=0, priority=0 actions=drop  

 cookie=0x4060000000000, duration=3575.949s, table=5, n_packets=82625967, n_bytes=23707729104, priority=200,ip actions=load:0x4->NXM_NX_REG0[0..15],resubmit(,30) 
 cookie=0x4000000000000, duration=3582.361s, table=5, n_packets=1366755, n_bytes=89993906, priority=0 actions=LOCAL 

 cookie=0x520, duration=2721.127s, table=6, n_packets=40, n_bytes=15524, ct_state=+inv+trk,ip actions=drop 
 cookie=0x520, duration=2742.719s, table=6, n_packets=3, n_bytes=198, priority=520,ct_state=+new+trk,ip actions=ct(commit,table=7,zone=64520)                     
 cookie=0x520, duration=2742.273s, table=6, n_packets=38, n_bytes=4709, priority=0 actions=resubmit(,7)  

 cookie=0x520, duration=2749.068s, table=7, n_packets=41, n_bytes=4907, priority=520,ip,reg0=0x10000/0x10000 actions=output:NXM_NX_REG1[] 
 cookie=0x520, duration=2749.025s, table=7, n_packets=0, n_bytes=0, priority=0 actions=drop

How to install tested OpenFlow rules

$GWOFPort=2
$UplinkOFPort=3
$PodOFPort=4
$LocalOFPort="LOCAL"

$PodIP="192.168.194.8"
$PodMac="00:15:5d:61:83:41"
$LocalPodCIDR="192.168.194.0/24"
$RemotePodCIDR="192.168.193.0/24"
$RemoteNodeIP="10.176.25.211"
$GWIP="192.168.192.1"
$GWMac="00:15:5d:1a:d4:0b"

$ClassifierTable="0"
$ContrackTable="1"
$ContrackStateTable="2"
$L3ForwardingTable="3"
$L2ForwardingCalcTable="4"
$ContrackCommitTable="6"
$L2ForwardingOutTable="7"

$Cookie="cookie=0x520"
$Priority="priority=520"
$CTZone="64520"

# Install Flows

## 0. Default flows
ovs-ofctl add-flow br-int "table=$L2ForwardingOutTable, $Cookie,priority=0 actions=drop"
ovs-ofctl add-flow br-int "table=$ContrackCommitTable, $Cookie,priority=0 actions=resubmit(,$L2ForwardingOutTable)"
ovs-ofctl add-flow br-int "table=$L2ForwardingCalcTable, $Cookie,priority=0 actions=drop"
ovs-ofctl add-flow br-int "table=$L3ForwardingTable, $Cookie,priority=0 ip, actions=dec_ttl,resubmit(,$L2ForwardingCalcTable)" 
ovs-ofctl add-flow br-int "table=$ContrackStateTable, $Cookie,priority=0 actions=drop"
ovs-ofctl add-flow br-int "table=$ContrackTable, $Cookie,priority=0 actions=drop"

## 1. request pkts
### Pod --> OVS --> GW --> HostNetwork
ovs-ofctl add-flow br-int "table=$ClassifierTable, $Cookie, $Priority, in_port=$PodOFPort, ip, nw_src=$PodIP, nw_dst=$RemotePodCIDR actions=resubmit(,$ContrackTable)" 
ovs-ofctl add-flow br-int "table=$ContrackTable, $Cookie, $Priority, ip, actions=ct(table=$ContrackStateTable,zone=$CTZone,nat)"
ovs-ofctl add-flow br-int "table=$ContrackStateTable, $Cookie, $Priority, ip actions=resubmit(,$L3ForwardingTable)" 
ovs-ofctl add-flow br-int "table=$L3ForwardingTable, $Cookie, $Priority, ip, nw_dst=$RemotePodCIDR actions=dec_ttl,mod_dl_dst:$GWMac,resubmit(,$L2ForwardingCalcTable)" 
ovs-ofctl add-flow br-int "table=$L2ForwardingCalcTable, $Cookie, $Priority, dl_dst=$GWMac actions=load:0x$GWOFPort->NXM_NX_REG1[],load:0x1->NXM_NX_REG0[16],resubmit(,$ContrackCommitTable)" # set gw as putport; mark as known
ovs-ofctl add-flow br-int "table=$ContrackCommitTable, $Cookie, $Priority, ct_state=+new+trk,ip actions=ct(commit,table=$L2ForwardingOutTable,zone=$CTZone)"
ovs-ofctl add-flow br-int "table=$L2ForwardingOutTable, $Cookie, $Priority, ip,reg0=0x10000/0x10000 actions=output:NXM_NX_REG1[]"
ovs-ofctl add-flow br-int "table=$L2ForwardingOutTable, $Cookie,priority=0 actions=drop"
### HostNetwork --> LOCAL --> OVS --> Uplink --> RemoteNode
ovs-ofctl add-flow br-int "table=$ClassifierTable, $Cookie, $Priority, in_port=$LocalOFPort, ip, nw_dst=$RemotePodCIDR actions=$UplinkOFPort"

## 2. reply pkts
### RemoteNode --> Uplink --> OVS --> LCOAL --> HostNetwork
ovs-ofctl add-flow br-int "table=$ClassifierTable, $Cookie, $Priority, in_port=$UplinkOFPort, ip, nw_src=$RemotePodCIDR actions=$LocalOFPort"
### HostNetwork --> GW --> OVS --> Pod
ovs-ofctl add-flow br-int "table=$ClassifierTable, $Cookie, $Priority, in_port=$GWOFPort, ip, nw_src=$RemotePodCIDR, nw_dst=$PodIP actions=resubmit(,$ContrackTable)" 
ovs-ofctl add-flow br-int "table=$L3ForwardingTable, $Cookie, ip, actions=dec_ttl,resubmit(,$L2ForwardingCalcTable)" 
ovs-ofctl add-flow br-int "table=$L2ForwardingCalcTable, $Cookie, $Priority, dl_dst=00:15:5d:61:83:41 actions=load:0x$PodOFPort->NXM_NX_REG1[],load:0x1->NXM_NX_REG0[16],resubmit(,$ContrackCommitTable)"
ovs-ofctl add-flow br-int "table=$ContrackCommitTable, $Cookie, ct_state=+inv+trk,ip actions=drop"
ovs-ofctl add-flow br-int "table=$ContrackCommitTable, $Cookie,priority=0 actions=resubmit(,$L2ForwardingOutTable)"

## 3. Install ARP responder flows for Pod

The topo and rules are comlicated, please feel free to ping me for more details.

commandgjj commented 3 years ago

Thanks Rui. I will check it more.