search

kytos-ng / kytos-end-to-end-tests

Kytos End-to-End Tests
0 stars 10 forks source link

helpers function to reconnect_switches lead to removal of all flows #188

Closed italovalcy closed 1 year ago

italovalcy commented 1 year ago

Hi,

I was troubleshooting the issues that occurred in the e2e below:

rerun: 0
tests/test_e2e_20_flow_manager.py::TestE2EFlowManager::test_040_replace_action_flow: 2022-12-01,06:11:21.855076 - 2022-12-01,06:11:46.884125
self = <tests.test_e2e_20_flow_manager.TestE2EFlowManager object at 0x7f36d4a82220>

   def test_040_replace_action_flow(self):
     self.replace_action_flow()

tests/test_e2e_20_flow_manager.py:446: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <tests.test_e2e_20_flow_manager.TestE2EFlowManager object at 0x7f36d4a82220>
restart_kytos = False

   def replace_action_flow(self, restart_kytos=False):

       payload = {
           "flows": [
               {
                   "priority": 10,
                   "idle_timeout": 360,
                   "hard_timeout": 1200,
                   "match": {
                       "in_port": 1
                   },
                   "actions": [
                       {
                           "action_type": "output",
                           "port": 2
                       }
                   ]
               }
           ]
       }

       api_url = KYTOS_API + '/flow_manager/v2/flows/00:00:00:00:00:00:00:01'
       requests.post(api_url, data=json.dumps(payload),
                     headers={'Content-type': 'application/json'})

       # wait for the flow to be installed
       time.sleep(10)

       # Verify the flow
       s1 = self.net.net.get('s1')
       flows_s1 = s1.dpctl('dump-flows')
       assert len(flows_s1.split('\r\n ')) == BASIC_FLOWS + 1, flows_s1
       assert 'in_port="s1-eth1' in flows_s1

       # Modify the actions and verify its modification
       s1.dpctl('mod-flows', 'actions=output:3')
       flows_s1 = s1.dpctl('dump-flows')
       assert 'actions=output:"s1-eth2"' not in flows_s1
       assert 'actions=output:"s1-eth3"' in flows_s1

       if restart_kytos:
           # restart controller keeping configuration
           self.net.start_controller(enable_all=True)
           self.net.wait_switches_connect()
       else:
           self.net.reconnect_switches()

       time.sleep(10)

       # Check that the flow keeps the original setting
       s1 = self.net.net.get('s1')
       flows_s1 = s1.dpctl('dump-flows')
     assert len(flows_s1.split('\r\n ')) == BASIC_FLOWS + 1, flows_s1
E       AssertionError:  cookie=0xab00000000000001, duration=14.026s, table=0, n_packets=8, n_bytes=336, priority=1000,dl_vlan=3799,dl_type=0x88cc actions=CONTROLLER:65535
E         
E       assert 1 == (3 + 1)
E        +  where 1 = len([' cookie=0xab00000000000001, duration=14.026s, table=0, n_packets=8, n_bytes=336, priority=1000,dl_vlan=3799,dl_type=0x88cc actions=CONTROLLER:65535\r\n'])
E        +    where [' cookie=0xab00000000000001, duration=14.026s, table=0, n_packets=8, n_bytes=336, priority=1000,dl_vlan=3799,dl_type=0x88cc actions=CONTROLLER:65535\r\n'] = <built-in method split of str object at 0x7f36b05bc100>('\r\n ')
E        +      where <built-in method split of str object at 0x7f36b05bc100> = ' cookie=0xab00000000000001, duration=14.026s, table=0, n_packets=8, n_bytes=336, priority=1000,dl_vlan=3799,dl_type=0x88cc actions=CONTROLLER:65535\r\n'.split

tests/test_e2e_20_flow_manager.py:441: AssertionError

One of the reasons why this error above happened was: once we ran the reconnect_switches function, all flows are removed.

See this discussion here: https://mail.openvswitch.org/pipermail/ovs-discuss/2014-May/033712.html

Proof of concept:

root@ee736f74a4a5:/kytos-end-to-end-tests# ovs-ofctl dump-flows s1
 cookie=0xac00000000000001, duration=26.644s, table=0, n_packets=0, n_bytes=0, priority=50000,dl_src=ee:ee:ee:ee:ee:02 actions=CONTROLLER:65535
 cookie=0xab00000000000001, duration=28.118s, table=0, n_packets=10, n_bytes=420, priority=1000,dl_vlan=3799,dl_type=0x88cc actions=CONTROLLER:65535
 cookie=0x0, duration=26.641s, table=0, n_packets=0, n_bytes=0, idle_timeout=360, hard_timeout=1200, priority=10,in_port="s1-eth1" actions=output:"s1-eth2"
root@ee736f74a4a5:/kytos-end-to-end-tests# ovs-vsctl del-controller s1; ovs-vsctl set-controller s1 tcp:127.0.0.1:6653; ovs-ofctl dump-flows s1
root@ee736f74a4a5:/kytos-end-to-end-tests# sleep 10
root@ee736f74a4a5:/kytos-end-to-end-tests# ovs-ofctl dump-flows s1
 cookie=0xac00000000000001, duration=19.088s, table=0, n_packets=0, n_bytes=0, priority=50000,dl_src=ee:ee:ee:ee:ee:02 actions=CONTROLLER:65535
 cookie=0xab00000000000001, duration=20.575s, table=0, n_packets=6, n_bytes=252, priority=1000,dl_vlan=3799,dl_type=0x88cc actions=CONTROLLER:65535
 cookie=0x0, duration=19.084s, table=0, n_packets=0, n_bytes=0, idle_timeout=360, hard_timeout=1200, priority=10,in_port="s1-eth1" actions=output:"s1-eth2"

The sleep shows that after some time the flows are recreated (due to the consistency check). The point is: right after changing the controller, the flows are removed.

I believe the expected behavior for reconnect_switches() is just to reset the connection to the controller (to force the consistency check to run). Removing the flows seems to be an unexpected behavior.

viniarck commented 1 year ago

@italovalcy thanks for looking into this. Yes, that's was an unexpected side effect.

I'll update the function to set to an unreachable port as they've suggested in this discussion. Locally explored it to confirm and indeed it behaved as expected only forcing the handshake again:

❯ sudo ovs-ofctl -O OpenFlow13 dump-flows s1
 cookie=0xac00000000000001, duration=104.822s, table=0, n_packets=0, n_bytes=0, send_flow_rem priority=50000,dl_src=ee:ee:ee:ee:ee:02 actions=CONTROLLER:65535
 cookie=0xac00000000000001, duration=104.821s, table=0, n_packets=0, n_bytes=0, send_flow_rem priority=50000,dl_src=ee:ee:ee:ee:ee:03 actions=CONTROLLER:65535
 cookie=0xaaa5949e826a664e, duration=153.887s, table=0, n_packets=4, n_bytes=280, send_flow_rem priority=5000,in_port="s1-eth1" actions=push_vlan:0x88a8,set_field:6157->vlan_vid,output:"
s1-eth4"
 cookie=0xaaa5949e826a664e, duration=153.883s, table=0, n_packets=4, n_bytes=296, send_flow_rem priority=5000,in_port="s1-eth4",dl_vlan=2061 actions=pop_vlan,output:"s1-eth1"
 cookie=0xaaa5949e826a664e, duration=153.740s, table=0, n_packets=0, n_bytes=0, send_flow_rem priority=5000,in_port="s1-eth2",dl_vlan=3738 actions=pop_vlan,output:"s1-eth1"
 cookie=0xab00000000000001, duration=83.109s, table=0, n_packets=110, n_bytes=4620, send_flow_rem priority=1000,dl_vlan=3799,dl_type=0x88cc actions=CONTROLLER:65535
kytos $> 2022-12-02 10:53:49,095 - INFO [kytos.core.atcp_server] (MainThread) Connection lost with client 127.0.0.1:40012. Reason: Request closed by client
2022-12-02 10:53:49,116 - INFO [kytos.core.atcp_server] (MainThread) New connection from 127.0.0.1:60880
2022-12-02 10:53:49,140 - INFO [kytos.napps.kytos/of_core] (thread_pool_sb_3) Connection ('127.0.0.1', 60880), Switch 00:00:00:00:00:00:00:01: OPENFLOW HANDSHAKE COMPLETE

❯ sudo ovs-vsctl set-controller s1 tcp:127.0.0.1:6666; sudo ovs-vsctl set-controller s1 tcp:127.0.0.1:6653; sudo ovs-ofctl dump-flows s1 -O OpenFlow13
 cookie=0xac00000000000001, duration=108.764s, table=0, n_packets=0, n_bytes=0, send_flow_rem priority=50000,dl_src=ee:ee:ee:ee:ee:02 actions=CONTROLLER:65535
 cookie=0xac00000000000001, duration=108.763s, table=0, n_packets=0, n_bytes=0, send_flow_rem priority=50000,dl_src=ee:ee:ee:ee:ee:03 actions=CONTROLLER:65535
 cookie=0xaaa5949e826a664e, duration=157.829s, table=0, n_packets=4, n_bytes=280, send_flow_rem priority=5000,in_port="s1-eth1" actions=push_vlan:0x88a8,set_field:6157->vlan_vid,output:"
s1-eth4"
 cookie=0xaaa5949e826a664e, duration=157.825s, table=0, n_packets=4, n_bytes=296, send_flow_rem priority=5000,in_port="s1-eth4",dl_vlan=2061 actions=pop_vlan,output:"s1-eth1"
 cookie=0xaaa5949e826a664e, duration=157.682s, table=0, n_packets=0, n_bytes=0, send_flow_rem priority=5000,in_port="s1-eth2",dl_vlan=3738 actions=pop_vlan,output:"s1-eth1"
 cookie=0xab00000000000001, duration=87.051s, table=0, n_packets=112, n_bytes=4704, send_flow_rem priority=1000,dl_vlan=3799,dl_type=0x88cc actions=CONTROLLER:65535