sdntrace is not working on complex topologies

italovalcy commented 8 months ago

Hi,

In a simple topology, like linear,3, sdntrace works fine. However, if you use a more complex topology with multiple links between switches, loops, etc, the trace just stops on the first switch.

How to reproduce?

Instantiate Kytos docker container amlight/kytos:latest and run Mininet with a custom topology [1] (tmux new-sess -d -s mn mn --custom /custom.py --topo mytopo --controller=remote,ip=127.0.0.1)
Enable all links, switches and interfaces
Create an EVC between s1 and s5 (vlan=400; curl -H 'Content-type: application/json' -X POST http://127.0.0.1:8181/api/kytos/mef_eline/v2/evc/ -d '{"name": "evc-vlan-'$vlan'", "dynamic_backup_path": true, "uni_a": {"tag": {"value": '$vlan', "tag_type": 1}, "interface_id": "00:00:00:00:00:00:00:01:16"}, "uni_z": {"tag": {"value": '$vlan', "tag_type": 1}, "interface_id": "00:00:00:00:00:00:00:05:16"}}')

Run a trace from s5:

root@e7599de2b1af:/# curl -X PUT -H 'Content-type: application/json' http://127.0.0.1:8181/api/amlight/sdntrace/trace -d '{"trace": {"switch": {"dpid": "00:00:00:00:00:00:00:05", "in_port": 16}, "eth": {"dl_vlan": 400}}}'
{"result":{"trace_id":30001}}
root@e7599de2b1af:/# curl -s http://127.0.0.1:8181/api/amlight/sdntrace/trace/30001 | jq -r
{
"request": {
"trace": {
  "eth": {
    "dl_vlan": 400
  },
  "switch": {
    "dpid": "00:00:00:00:00:00:00:05",
    "in_port": 16
  }
}
},
"request_id": 30001,
"result": [
{
  "dpid": "00:00:00:00:00:00:00:05",
  "port": 16,
  "time": "2024-03-11 20:16:03.487688",
  "type": "starting"
},
{
  "msg": "none",
  "reason": "done",
  "time": "0:00:01.520061",
  "type": "last"
}
],
"start_time": "2024-03-11 20:16:03.487688",
"total_time": "0:00:01.520511"
}

Expected result: Trace should go from s5 all the way to s1.

Actual result: as shown above, the trace stops on s5.

If you look into the Packet capture from the trace above, you will see the source MAC address does not match with the address that was supposed to be according to kytos-ng/coloring:

root@e7599de2b1af:/# tshark -i lo -Y "openflow_v4.type==13 and openflow_v4.packet_out.in_port==16" -V
...
OpenFlow 1.3
    Version: 1.3 (0x04)
    Type: OFPT_PACKET_OUT (13)
    Length: 170
    Transaction ID: 2886786446
    Buffer ID: OFP_NO_BUFFER (4294967295)
    In port: 16
    Actions length: 16
    Pad: 000000000000
    Action
        Type: OFPAT_OUTPUT (0)
        Length: 16
        Port: OFPP_TABLE (4294967289)
        Max length: OFPCML_NO_BUFFER (65535)
        Pad: 000000000000
    Data
        Ethernet II, Src: ee:ee:ee:ee:ee:05 (ee:ee:ee:ee:ee:05), Dst: ca:fe:ca:fe:ca:fe (ca:fe:ca:fe:ca:fe)
        802.1Q Virtual LAN, PRI: 0, DEI: 0, ID: 400
        Internet Protocol Version 4, Src: 1.1.1.1, Dst: 1.1.1.2
...

On the other hand, the only coloring flows s5 have are:

# ovs-ofctl dump-flows s5
 cookie=0xac00000000000005, duration=1178.079s, table=0, n_packets=0, n_bytes=0, priority=50000,dl_src=ee:ee:ee:ee:ee:06 actions=CONTROLLER:65535
 cookie=0xac00000000000005, duration=1178.077s, table=0, n_packets=0, n_bytes=0, priority=50000,dl_src=ee:ee:ee:ee:ee:02 actions=CONTROLLER:65535
 cookie=0xac00000000000005, duration=1178.041s, table=0, n_packets=0, n_bytes=0, priority=50000,dl_src=ee:ee:ee:ee:ee:03 actions=CONTROLLER:65535

[1] Mininet custom topology

from mininet.topo import Topo

class MyTopo(Topo):
    def build(self):
        # Create two hosts
        h1 = self.addHost('h1', ip='0.0.0.0')
        h2 = self.addHost('h2', ip='0.0.0.0')
        h3 = self.addHost('h3', ip='0.0.0.0')
        h4 = self.addHost('h4', ip='0.0.0.0')
        h5 = self.addHost('h5', ip='0.0.0.0')
        h6 = self.addHost('h6', ip='0.0.0.0')
        # Create the switches
        s1 = self.addSwitch('s1')
        s2 = self.addSwitch('s2')
        s3 = self.addSwitch('s3')
        s4 = self.addSwitch('s4')
        s5 = self.addSwitch('s5')
        s6 = self.addSwitch('s6')
        # Add links between the switch and each host
        self.addLink(s1, h1, port1=16)
        self.addLink(s2, h2, port1=16)
        self.addLink(s3, h3, port1=16)
        self.addLink(s4, h4, port1=16)
        self.addLink(s5, h5, port1=16)
        self.addLink(s6, h6, port1=16)
        # Add links between the switches
        self.addLink(s2, s6, port1=1, port2=5)
        self.addLink(s5, s6, port1=8, port2=8)
        self.addLink(s5, s6, port1=9, port2=9)
        self.addLink(s6, s6, port1=12, port2=13)
        self.addLink(s6, s6, port1=25, port2=26)
        self.addLink(s2, s5, port1=3, port2=3)
        self.addLink(s3, s5, port1=6, port2=7)
        self.addLink(s5, s5, port1=19, port2=20)
        self.addLink(s2, s3, port1=4, port2=4)
        self.addLink(s2, s3, port1=10, port2=10)
        self.addLink(s3, s3, port1=12, port2=13)

# You can run any of the topologies above by doing:
# mn --custom /custom.py --topo ring --controller=remote,ip=127.0.0.1
topos = {
    'mytopo': (lambda: MyTopo()),
}

italovalcy commented 8 months ago

Update: the payload I was sending is incorrect. Thus, this basic test is passing. Another possible cause of an issue similar to this one is on Noviflow devices, which are not running the trace correctly (as if the action TABLE were not working)

italovalcy commented 8 months ago

Confirmed the problem is on Noviflow side, not supporting correctly the action OFPP_TABLE depending on the pipeline config.

viniarck commented 8 months ago

I suspect it's the same as https://github.com/kytos-ng/sdntrace/issues/58, in that issue with an EVPL during a trace it was as if a drop was happening, but the back end it wasn't completely investigated. It might be worth to reassess if the priority and the target of the issue needs to be reassessed @italovalcy

kytos-ng / sdntrace

sdntrace is not working on complex topologies #63