intervlan routing with multiple dp connected with trunks does not work

[kstest1]

I have 3 dp: s1, s2, s3. Configured vlans: 100 and 200. Configured router on vl100 and vl200. Switch connections: s1, port 20 (of port1 -see config) connects to s2, port 20 (of port1) . vlan 100 and 200 allowed with tagging. s2, port 21 (of port2) connects to s3, port 21. (of port2) vlan 100 and 200 allowed with tagging. Host connections: On s1, port 22 (of port3) , there is host h1 in native vlan 100 ( ip 10.0.100.51/24, d.g. 10.0.100.254) On s3, port 22 (of port3) , there is host h2 in native vlan 200.( ip 10.0.200.52/24, d.g. 10.0.200.254) On s3, port 20 (of port1) , there is host h3 in native vlan 100.( ip 10.0.100.53/24, d.g. 10.0.100.254) Result: h1 can ping h3 in same vlan. h2 can ping h3 in different vlan ( they are on s3 switch). h1 can NOT ping h2 in different vlan ( they are on s1 /s3 switches connected via s2).

Reason: When h2 pings h1, s3 needs to know arp info for h1 in vlan 100. arp is generated from faucet and sent from s3 , port 21 to s2, port21. This packet is dropped on s2 as came from faucet vip dl address. Even if it is allowed by config, return packet ARP will be taken by s1 as reply go to same mac address and s3 will not know mapping.

Config ( test with hardware: "GenericTFM", same with other hw)

test#more /etc/faucet/faucet.yaml 
include:
   - acls.yaml

vlans:
    office:
        vid: 100
        description: "office network"
        faucet_mac: "0e:00:00:00:10:01"
        faucet_vips: ['10.0.100.254/24', '2001:100::1/64', 'fe80::c00:00ff:fe00:1001/64']
    guest:
        vid: 200
        description: "guest network"
        faucet_mac: "0e:00:00:00:20:01"
        faucet_vips: ['10.0.200.254/24', '2001:200::1/64', 'fe80::c00:00ff:fe00:2001/64']

routers:
    router-office-guest:
        vlans: [office, guest]

dps:
    sw1:
        dp_id: 0x1
        hardware: "GenericTFM"
        interfaces:
            3:
                name: "if22"
                description: "if22-offvl"
                native_vlan: office
            1:
                name: "if20"
                description: "trunk to sw2"
                tagged_vlans: [office, guest]

    sw2:
        dp_id: 0x2
        hardware: "GenericTFM"
        interfaces:
            1:
                name: "if20"
                description: "trunk to sw1"
                tagged_vlans: [office, guest]

            2:
                name: "if21"
                description: "trunk to sw3"
                tagged_vlans: [office, guest]

    sw3:
        dp_id: 0x3
        hardware: "GenericTFM"
        interfaces:

            2:
                name: "if21"
                description: "trunk to sw2"
                tagged_vlans: [office, guest]

            3:
                name: "if22"
                description: "if22-guestvl"
                native_vlan: guest

            1:
                name: "if20"
                description: "if22-offvl"
                native_vlan: office

Tables 0 and 1 flows on s2

OFPST_FLOW reply (OF1.3) (xid=0x2): cookie=0x5adc15c0, duration=18.748s, table=0, n_packets=0, n_bytes=0, priority=9000,in_port=1,dl_vlan=100 actions=goto_table:1 cookie=0x5adc15c0, duration=18.748s, table=0, n_packets=0, n_bytes=0, priority=9000,in_port=1,dl_vlan=200 actions=goto_table:1 cookie=0x5adc15c0, duration=18.747s, table=0, n_packets=2, n_bytes=120, priority=9000,in_port=2,dl_vlan=100 actions=goto_table:1 cookie=0x5adc15c0, duration=18.747s, table=0, n_packets=0, n_bytes=0, priority=9000,in_port=2,dl_vlan=200 actions=goto_table:1 cookie=0x5adc15c0, duration=18.747s, table=0, n_packets=0, n_bytes=0, priority=0 actions=drop cookie=0x5adc15c0, duration=18.751s, table=1, n_packets=0, n_bytes=0, priority=9100,dl_src=ff:ff:ff:ff:ff:ff actions=drop cookie=0x5adc15c0, duration=18.751s, table=1, n_packets=0, n_bytes=0, priority=9100,dl_src=0e:00:00:00:20:01 actions=drop cookie=0x5adc15c0, duration=18.750s, table=1, n_packets=2, n_bytes=120, priority=9100,dl_src=0e:00:00:00:10:01 actions=drop cookie=0x5adc15c0, duration=18.751s, table=1, n_packets=0, n_bytes=0, priority=9100,dl_type=0x9000 actions=drop cookie=0x5adc15c0, duration=18.750s, table=1, n_packets=0, n_bytes=0, priority=9099,arp,dl_vlan=200 actions=goto_table:4 cookie=0x5adc15c0, duration=18.750s, table=1, n_packets=0, n_bytes=0, priority=9099,arp,dl_vlan=100 actions=goto_table:4 cookie=0x5adc15c0, duration=18.750s, table=1, n_packets=0, n_bytes=0, priority=9099,ip,dl_vlan=200,dl_dst=0e:00:00:00:20:01 actions=goto_table:2 cookie=0x5adc15c0, duration=18.750s, table=1, n_packets=0, n_bytes=0, priority=9099,ipv6,dl_vlan=200,dl_dst=33:33:ff:00:00:01 actions=goto_table:4 cookie=0x5adc15c0, duration=18.750s, table=1, n_packets=0, n_bytes=0, priority=9099,ipv6,dl_vlan=200,dl_dst=0e:00:00:00:20:01 actions=goto_table:3 cookie=0x5adc15c0, duration=18.750s, table=1, n_packets=0, n_bytes=0, priority=9099,ipv6,dl_vlan=200,dl_dst=33:33:00:00:00:02 actions=goto_table:4 cookie=0x5adc15c0, duration=18.750s, table=1, n_packets=0, n_bytes=0, priority=9099,ipv6,dl_vlan=200,dl_dst=33:33:ff:00:20:01 actions=goto_table:4 cookie=0x5adc15c0, duration=18.750s, table=1, n_packets=0, n_bytes=0, priority=9099,ip,dl_vlan=100,dl_dst=0e:00:00:00:10:01 actions=goto_table:2 cookie=0x5adc15c0, duration=18.750s, table=1, n_packets=0, n_bytes=0, priority=9099,ipv6,dl_vlan=100,dl_dst=33:33:ff:00:00:01 actions=goto_table:4 cookie=0x5adc15c0, duration=18.750s, table=1, n_packets=0, n_bytes=0, priority=9099,ipv6,dl_vlan=100,dl_dst=0e:00:00:00:10:01 actions=goto_table:3 cookie=0x5adc15c0, duration=18.750s, table=1, n_packets=0, n_bytes=0, priority=9099,ipv6,dl_vlan=100,dl_dst=33:33:00:00:00:02 actions=goto_table:4 cookie=0x5adc15c0, duration=18.750s, table=1, n_packets=0, n_bytes=0, priority=9099,ipv6,dl_vlan=100,dl_dst=33:33:ff:00:10:01 actions=goto_table:4 cookie=0x5adc15c0, duration=18.749s, table=1, n_packets=0, n_bytes=0, priority=9000,dl_vlan=200 actions=CONTROLLER:96,goto_table:5 cookie=0x5adc15c0, duration=18.748s, table=1, n_packets=0, n_bytes=0, priority=9000,dl_vlan=100 actions=CONTROLLER:96,goto_table:5 cookie=0x5adc15c0, duration=18.747s, table=1, n_packets=0, n_bytes=0, priority=0 actions=goto_table:5

[anarkiwi]

You will need to change faucet_mac on one of the DPs to a different value (see https://docs.faucet.nz/en/latest/configuration.html#faucet-configuration).

Please re-open if this does not fix the problem.

[KitL]

I want to reopen this because I think this is a bad behaviour to have occurring by default. So we should definitely fix this somehow.

Also I am not 100% sure how you go about configuring different macs on different dps. Is it still the case that you configure two separate vlans with the same vid, one is used for routing and the others for switching? Again that is a horrible way of configuring things, and not trivial to interpret from the documentation.

I think a better solution might be to add drop_spoofed_faucet_mac: False on all the dps that dont contain routers, though I am not certain that wont cause other issues. If it does work perhaps faucet should just enable that by default?

[gizmoguy]

Setting drop_spoofed_faucet_mac to false on DPs that don't have routing enabled sounds sensible to me.

You are right, we do need a better way of handling multi-DP routing, currently the workaround is to have 2 controllers (an L3 controller and an L2 controller) with separate configuration files that specify different VLANs with the same VIDs (one with L3 configuration and one without). This is not documented anywhere and could be made more user friendly, unfortunately no good ideas come to mind right now on how to fix this. Possibly we could fix this by pushing distributed switching more, however we don't have L3 support in the distributed switching code so that is no-go for now.

[kstest1]

You will need to change faucet_mac on one of the DPs to a different value (see https://docs.faucet.nz/en/latest/configuration.html#faucet-configuration).

Please re-open if this does not fix the problem.

faucet_mac per documentation is vlan attribute not DP attribute so I do not know how to tie faucet_mac to specific DP. Regarding KitL comment, do we have to have different vlan name with same 802.1q vlan id and use different names on different switches? E.g on sw1, should ports in vl 100 be assigned to vlan "vl100a" and on sw3 to "vl100c" - both names, same VID 100, different mac and same VIP address?

[gizmoguy]

You are correct you can't tie faucet_mac to a DP, it is applied at the VLAN level.

Since faucet doesn't let you reuse VIDs between different VLANs you end up needing 2 separate instances of faucet running different config files to get the same VIDs reused on VLANs with different configuration.

There is a slightly complicated example of how to do this here:

https://github.com/wandsdn/sc18-faucet-configs

Sorry this isn't as easy as it could be and we'll try think of ways to make this easier for you.

[kstest1]

OK, now is clear. From github example, it is obvious that L2 without L3 works fine in topology with multiple switches and L3 works on one switch so you used "L3 switch on the stick", controlled with different controller than L2 domain. I did not examine source code but probably current architecture do not cover case of "integrated L2 and L3" on arbitrary topology. My hint how to go next (regarding your "we'll try think of ways to make this ") is to make something similar to e.g. Cisco Anycast HSRP used with Cisco Fabric path or distributed router seen on Openstack. Basic idea is that you have one IP address and one MAC address - not multiple MAC addresses. Packet sent to this MAC address, must be intercepted on every switch and must be routed ( so we have distributed routing). In my example, when h2 pings h1, h2 send packet to VIP mac in vl200. s3 process packet but to route them, there should be entry on s3 with destination MAC of h1 in vl100. As there is no entry, ARP is sent by controller to s3 ( seen in wireshark in packet out message) and s3 send this ARP packet to s2. Here, s2 drops packet. If I disable drop_spoofed_faucet_mac on s1, s2, ARP packet will eventually come to h1 and h1 will respond. But, response will be caught by s1 (dest mac is vip mac in vl 100), and probably will not go to s3. Solution will be foward this information from s1 to controller and reuse this information on controller for s3 (install appropriate flows on s3) .

[githubassumde]

This is related I think:

I have a simple OVS topology to test if Faucet can support our infrastructure (one spine. several leafs, VLANs spread amongst leafs):

SW1 - trunk - SW2 - h1 & h2

The trunk is configured for tagged VLANs 100 and 200, h1 connects untagged to SW2 and VLAN 100, h2 connects untagged to SW2 and VLAN 200

SW1 and SW2 each have their own Faucet server (to separate L2 and L3, as in gizmoguys example).

Faucet for SW1 has a router configured for VLANs 100 and 200. It also has faucet_mac and faucet_vips configured for each VLAN.

Faucet for SW2 just distributes the VLANs.

I attached Faucet configuration for SW1 and SW2 and the commands used for setting up the topology. Setup commands are based on the shell functions from the tutorial.

faucet_sw1.yaml.txt faucet_sw2.yaml.txt lab_setup.sh.txt

Problem: Hosts h1 and h2 cannot communicate.

The reason for this seems to be that OpenFlow in general permits output to ingress port. From what I have read (unable to find it right now), this is by design in OpenFlow.

If I remove the trunk and insert two separate links between SW1 and SW2- one for each VLAN - connections re working. This seems to confirm the above.

Inspecting Flows on SW1 I see rewritings for fields vlan_vid, eth_src eth_dst. I think rewriting the ingress port can be a solution but I failed to do that with an ACL.

Can that be implemented and can I test this with an acl?

[gizmoguy]

We think this should work now, see this tutorial for how to set up our new Inter-VLAN routing between switches:

https://docs.faucet.nz/en/latest/tutorials/stacking.html#inter-vlan-routing-with-stacking

The usability issue of having to set drop_spoofed_faucet_mac to False is still there, I have #3172 open to address that problem.

Closing for now as I think this issue is now fixed, if not open a new ticket - thanks.