Closed vincentbernat closed 1 year ago
@chiragshah6 @sworleys @taspelund EVPN experts any idea here?
This looks to me like a Downstream-allocated VNI (D-VNI) setup (trying to install a route via the VNI in the route rather than via the L3VNI locally configured for the VRF) but you're using Traditional VXLAN Devices rather than a Single VXLAN Device.
@sworleys do you have an example iproute2 config they can try? i.e. the proper combo of ip link add vx0 type vxlan external vnifilter ...
and bridge vni add ...
(and possibly bridge vlan tunnel ...
) commands
Not a lot of documentation about single VXLAN devices, except Cumulus and ifupdown2, but I only see this used in conjunction with VLAN-aware bridges. No word on VRF, but maybe I need one VXLAN device per-VRF?
Aside from Cumulus and ifupdown2 docs, I don't really have any places to point you to for the linux interfaces needed to make EVPN go... I wrote the FRR doc for traditional bridges and traditional vxlan devices, but I haven't expanded them to cover vlan-aware bridges and single vxlan devices yet.
At a high level, to setup an SVD for EVPN you need to:
Example commands:
ip link add br0 type bridge vlan_filtering 1; ip link add name br0.10 link br0 type vlan id 10 protocol 802.1q; bridge vlan add vid 10 dev br0 self; ip link set eth0 master br0; bridge vlan add dev eth0 vid 10 master
ip link add vx0 type vxlan external vnifilter local <source-ip> dstport 4789 nolearning
bridge vni add dev vx0 vni 200010
ip link set vx0 master br0
ip link set vx0 type bridge_slave vlan_tunnel on
bridge vlan add dev vx0 vid 10 master
bridge vlan add dev vx0 vid 10 tunnel_info id 200010 master
That would be a good example for an L2VNI, since the VLAN is being extended to a physical/host-facing port (eth0). For an L3VNI you'd want to create a VRF device, set that VRF as the SVI's master device, and follow the same process as above minus adding physical ports to the corresponding VLAN (the vxlan device should be the only bridge-port carrying the L3VNI VLAN).
e.g.
ip link add TENANT1 type vrf id 2222
ip link add name br0.4000 link br0 type vlan id 4000 protocol 802.1q; bridge vlan add vid 4000 dev br0 self; ip link set br0.4000 master TENANT1
bridge vni add dev vx0 vni 204000
bridge vlan add dev vx0 vid 4000 master
bridge vlan add dev vx0 vid 4000 tunnel_info id 204000 master
No word on VRF, but maybe I need one VXLAN device per-VRF?
Currently FRR treats all VXLAN devices only as L2 interfaces, so they would decap and bridge traffic to a local bridge-port or SVI. Conceptually you could use a VXLAN device as an L3 interface, but FRR doesn't have code to support that yet. Today for an L3VNI you would need an SVI for the VRF that lives in the L3VNI VLAN (in the above example: VRF=TENANT1, L3VNI=204000, L3VNI-VLAN=4000, L3-SVI=br0.4000). Also remember to configure vrf TENANT1
+ vni 204000
in FRR so it knows this VNI should be L3 instead of L2.
Thanks for the answer. I am now using this setup: https://github.com/vincentbernat/network-lab/blob/master/lab-frr-evpn-vrf/setup.
From FRR point of view, everything looks OK:
R2# show ip route vrf vrf2
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
VRF vrf2:
B>* 10.0.10.0/24 [200/0] via 100.64.0.1, vx0 (vrf default) onlink, label 100, weight 1, 00:03:08
B>* 10.0.10.1/32 [200/0] via 100.64.0.1, vx0 (vrf default) onlink, label 100, weight 1, 00:03:08
C>* 10.0.20.0/24 is directly connected, l2vni220, 00:03:10
10.0.10.0/24 which originated in vrf1
on first host is available in vrf2
in second host, with the right label (100). On the kernel side, it seems to be correctly translated:
10.0.10.0/24 nhid 29 encap ip id 100 src 0.0.0.0 dst 100.64.0.1 ttl 0 tos 0 via 100.64.0.1 dev vx0 proto bgp metric 20 onlink
And in practice, this works!
Describe the bug
When importing several RT (different L3VNI for example) inside a VRF, the corresponding routes are installed with the nexthop of the L3VNI associated to the VRF instead of the nexthop of the VNI associated with the route.
This does not happen exactly on master because since b991a37262539cda53b6828f1ce993b74f1f9817, the route is considered inactive, but still with the wrong nexthop. This is fully reproducible with 8.5.2.
To Reproduce
Two hosts are directly connected to each other and establish a BGP session with L2VPN EVPN family only.
Host A:
Host B has the exact same configuration except the router ID and the neighbor address.
On host A, we have
192.0.2.46/28
in VRFA (containing a bridge with the address and a VXLAN interface enslaved). On host B, we have192.0.2.62/28
in VRFB (containing a bridge with the address and a VXLAN interface enslaved). You can see the full setup here: https://github.com/vincentbernat/network-lab/blob/master/lab-frr-evpn-vrf/setup.On host A, I see:
I would have expected to use
bridgeB
as the nexthop, because192.0.2.48/28
has VNI 80002:This also happens for L2VNI (but I don't have a minimal reproduction). I am not sure if this is expected to work (but in this case, I don't see a use for import/export RT).
Versions
cc @ton31337 (not the bug I talked about you, this one is gone in 8.5.1, so I didn't investigate).