Invalid traffic routing for host

[agowa]

Return traffic arrives on a different interface than the requests are sent with. Return traffic arrives from the physical interface, and outbound traffic is sent using the internal interface. This surprisingly doesn't cause issues with a bunch of use cases like ping, but breaks with others like mDNS or just tcpdump.

vswitch:

# ovs-vsctl show
77d884ae-c4db-4281-9ba2-1624fb41c7c3
    Bridge ovsbr0
        Port vlan20
            tag: 20
            Interface vlan20
                type: internal
        Port bond0
            tag: 20
            Interface bond0
        Port ovsbr0
            tag: 9
            Interface ovsbr0
                type: internal

# ovs-ofctl dump-ports-desc ovsbr0
OFPST_PORT_DESC reply (xid=0x2):
 3(bond0): addr:a8:03:98:af:c1:18
     config:     0
     state:      STP_FORWARD
     speed: 0 Mbps now, 0 Mbps max
 4(vlan20): addr:e8:cb:e9:97:23:bc
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max
 LOCAL(ovsbr0): addr:c9:f6:0e:58:83:9b
     config:     0
     state:      0
     speed: 0 Mbps now, 0 Mbps max

networkd config:

[Match]
Name=vlan*

[Network]
DHCP=yes
MulticastDNS=yes

• bond0 is a mode-0 I.E. balance-rr bridge created with iproute2's ip link using two slave MLX ports, which are directly connected to another computer (alpine linux)
• Openvswitch is configured on ArchLinux (with KDE GUI and libvirt KVM), kernel version 6.4.6-arch1-1

libvirt network definition:

<network>
  <name>ovs-net</name>
  <forward mode="bridge"/>
  <bridge name="ovsbr0"/>
  <virtualport type="openvswitch"/>
  <portgroup name="v020">
    <vlan trunk="yes">
      <tag id="10"/>
      <tag id="20"/>
    </vlan>
  </portgroup>
</network>

libvirt VM interface definition:

<interface type="network">
  <mac address="cd:4b:1c:fc:cb:1f"/>
  <source network="default" portid="..." bridge="virbr0"/>
  <target dev="vnet5"/>
  <model type="virtio"/>
  <alias name="net0"/>
  <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</interface>

Steps to reproduce:

1. Configure ovs as seen above
2. tcpdump/wireshark ovsbr0
3. make some traffic, ping or mDNS lookup resolvectl query "something.local" something externally connected through bond0
4. redo step 2-3 with vlan20 and bond0
5. Create a KVM/QEMU VM or LXC container and redo step 2-3.

Observation:

• mDNS is sent out via vlan20, replies are received on bond0 and also visible on ovsbr0, but never arrive at vlan20.
• ping reports successful replies in the terminal, but tcpdump/wireshark doesn't show return traffic. Somehow ping discovers and uses the return traffic directly from the bond0 interface.
• Within the VM/container, everything works as expected, BUT not for the host itself...