eBGP L2VPN EVPN route are not propagate by spine when NextHop is the same as the eBGP peer

shujaaa commented 1 month ago

Description

This takes place in a bgp/evpn vxlan fabric with spine-leaf architecture and multiple-as e-bgp (each leaf and the spine have different AS).

When a leaf uses the same loopback address for bgp evpn peering AND vxlan next hop, then the spine does not propagate the reiceived T2/T3/T5 routes to the other leafs.

When the leaf uses a separate loopback for eBGP peering and for VxLAN interface source (and nextHop) then the routes are correctly propagated by the spine.

Version

FRRouting 8.4.3 (leaf-2) on Linux(5.10.0-cl-1-amd64).

How to reproduce

Leaf-2 is cumulus/FRR swp1 goes to spine-1 Current configuration: ! frr version 8.4.3 frr defaults datacenter hostname leaf-2 log syslog informational service integrated-vtysh-config ! router bgp 65106 bgp router-id 172.30.0.6 neighbor underlay peer-group neighbor underlay remote-as external neighbor underlay advertisement-interval 0 neighbor underlay timers 3 9 neighbor underlay timers connect 10 neighbor swp1 interface peer-group underlay neighbor swp1 advertisement-interval 0 neighbor swp1 timers 3 9 neighbor swp1 timers connect 10 ! address-family ipv4 unicast redistribute connected maximum-paths 64 maximum-paths ibgp 64 exit-address-family ! address-family l2vpn evpn neighbor underlay activate advertise-all-vni vni 165 rd 172.30.0.6:165 route-target import 11:165 route-target export 11:165 exit-vni advertise-svi-ip exit-address-family exit !

Spine is cumulus/FRR, swp6 goes to leaf-2 frr version 8.4.3 frr defaults datacenter hostname spine-1 log syslog informational service integrated-vtysh-config ! router bgp 65100 bgp router-id 172.30.0.3 neighbor underlay peer-group neighbor underlay remote-as external neighbor underlay advertisement-interval 0 neighbor underlay timers 3 9 neighbor underlay timers connect 10 neighbor swp6 interface peer-group underlay neighbor swp6 advertisement-interval 0 neighbor swp6 timers 3 9 neighbor swp6 timers connect 10 neighbor 172.30.0.7 peer-group underlay neighbor 172.30.0.7 update-source 172.30.0.3 neighbor 172.30.0.7 advertisement-interval 0 neighbor 172.30.0.7 timers 3 9 neighbor 172.30.0.7 timers connect 10 neighbor 172.32.0.3 peer-group underlay neighbor 172.32.0.3 advertisement-interval 0 neighbor 172.32.0.3 timers 3 9 neighbor 172.32.0.3 timers connect 10 ! address-family ipv4 unicast redistribute connected maximum-paths 64 maximum-paths ibgp 64 exit-address-family ! address-family l2vpn evpn neighbor underlay activate advertise-all-vni exit-address-family exit !

the next leaf is runnning on an HPE Aruba switch in our example but can be reproduced with another cumulus/FRR machine, the key aspect is whether or not the vxlan source address is the same as the bgp-evpn neighbor address. evpn vlan 165 rd 172.30.0.7:165 route-target export 11:165 route-target import 11:165 interface loopback 0 ip address 172.30.0.7/32 interface vlan 165 vrf attach LR1 ip address 10.162.0.53/20 interface vxlan 1 source ip 172.30.0.7 no shutdown vni 2 vlan 2 vni 165 vlan 165 router bgp 65107 bgp router-id 172.32.0.3 bgp deterministic-med neighbor 172.30.0.3 remote-as 65100 neighbor 172.30.0.3 ebgp-multihop 5 neighbor 172.30.0.3 update-source 172.30.0.7 neighbor 172.32.0.1 remote-as 65100 address-family ipv4 unicast neighbor 172.32.0.1 activate redistribute connected redistribute local loopback exit-address-family address-family l2vpn evpn neighbor 172.30.0.3 send-community extended neighbor 172.30.0.3 activate exit-address-family !

"show bgp l2vpn evpn" on the spine will display the routes sent by 172.30.0.7. The same command on leaf-2 does not. when looking at routes received/advertised, spine-1 will show that it does not advertise 172.30.0.7 routes to leaf-2.

Expected behavior

All routes from neighbor 172.30.0.7 should be advertised to leaf-2

When changing the 172.30.0.7 config to the following interface loopback 0 ip address 172.30.0.7/32 interface loopback 1 ip address 172.31.0.4/32 interface vxlan 1 source ip 172.31.0.4

It works as expected.

No perceivable difference in the routes when changing the next hop IP address.

Expected when it works (with a different next hop) leaf-2# show bgp l2vpn evpn rd 172.30.0.7:165 BGP table version is 37, local router ID is 172.30.0.6 Status codes: s suppressed, d damped, h history, * valid, > best, i - internal Origin codes: i - IGP, e - EGP, ? - incomplete EVPN type-1 prefix: [1]:[EthTag]:[ESI]:[IPlen]:[VTEP-IP]:[Frag-id] EVPN type-2 prefix: [2]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP] EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP] EVPN type-4 prefix: [4]:[ESI]:[IPlen]:[OrigIP] EVPN type-5 prefix: [5]:[EthTag]:[IPlen]:[IP]

Network Next Hop Metric LocPrf Weight Path Route Distinguisher: 172.30.0.7:165

            172.31.0.4 (leaf-1)
                                                   0 65105 65100 65107 ?
            RT:11:165 ET:8

*> 172.31.0.4 (spine-1) 0 65100 65107 ? RT:11:165 ET:8

            172.31.0.4 (leaf-1)
                                                   0 65105 65100 65107 ?
            RT:11:165 ET:8

*> 172.31.0.4 (spine-1) 0 65100 65107 ? RT:11:165 ET:8

Displayed 2 out of 4 total prefixes

Actual behavior

leaf-2# show bgp l2vpn evpn rd 172.30.0.7:165 No prefixes displayed, 0 exist

Additional context

No response

Checklist

[X] I have searched the open issues for this bug.
[X] I have not included sensitive information in this report.

ton31337 commented 1 month ago

8.4.3 is an unsupported version anymore, can you test recent versions (9.1, 10.0)?

chiragshah6 commented 2 weeks ago

@shujaaa Can you try following config for numbered neighbor over multihop neighbor 172.30.0.7 disable-connected-check

FRRouting / frr