search

FRRouting / frr

The FRRouting Protocol Suite
https://frrouting.org/
Other
3.33k stars 1.25k forks source link

Not all routes removed using addpath-tx-all-paths with route reflectors #5500

Closed Try75 closed 4 years ago

Try75 commented 4 years ago

Describe the bug With FRR 6.x using addpath-tx-all-paths with redundant route reflectors, removing a network advertisement from one AS withdrew the advertisement from another AS that learned it. In FRR 7.2 with identical configurations, all of the duplicate routes from different addpath IDs are not withdrawn. It doesn't matter what version of FRR is used on the peers of the route reflectors, but with version 7.2 running on the route reflectors, incorrect behavior is observed.

FRR 6.x and 7.2 show ip bgp command shows all of duplicate routes learned by AS 4238440709:

cgw-4-east1# show ip bgp
BGP table version is 4, local router ID is 198.18.0.7, vrf id 0
Default local pref 100, local AS 4238440709
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 172.16.2.0/24    192.168.254.5                          0 65001 i
* i172.16.3.0/24    198.18.0.1               0    100      0 65001 i
* i                 198.18.0.12              0    100      0 65001 i
*>i                 198.18.0.1               0    100      0 65001 i
*=i                 198.18.0.12              0    100      0 65001 i

Displayed  2 routes and 5 total paths

After removing the advertisement of 172.16.3.0/24 from AS 65001 with FRR 6.x route reflectors, the routes are removed:

cgw-4-east1# show ip bgp
BGP table version is 3, local router ID is 198.18.0.7, vrf id 0
Default local pref 100, local AS 4238440709
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 172.16.2.0/24    192.168.254.5                          0 65001 i

Displayed  1 routes and 1 total paths

Replacing FRR 6.x with FRR 7.2 on the route reflectors only, using the same configuration, then removing the advertisement of 172.16.3.0/24 from AS 65001 only withdraws some of the routes:

cgw-4-east1# show ip bgp
BGP table version is 5, local router ID is 198.18.0.7, vrf id 0
Default local pref 100, local AS 4238440709
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 172.16.2.0/24    192.168.254.5                          0 65001 i
* i172.16.3.0/24    198.18.0.1               0    100      0 65001 i
*>i                 198.18.0.1               0    100      0 65001 i

Displayed  2 routes and 3 total paths

Output of frr.log in FRR 6.x with debug bgp update on cgw-4-east1: adding network 172.16.3.0/24 remotely

2019/12/07 02:05:58.208 BGP: 198.18.0.16 rcvd UPDATE w/ attr: nexthop 198.18.0.12, origin i, localpref 100, metric 0, originator 198.18.0.12, clusterlist 198.18.0.16, path 65001
2019/12/07 02:05:58.208 BGP: 198.18.0.16 rcvd UPDATE wlen 0 attrlen 49 alen 8
2019/12/07 02:05:58.208 BGP: 198.18.0.16 rcvd 172.16.3.0/24 with addpath ID 3 IPv4 unicast
2019/12/07 02:05:58.210 BGP: 198.18.0.16 rcvd UPDATE w/ attr: nexthop 198.18.0.1, origin i, localpref 100, metric 0, originator 198.18.0.1, clusterlist 198.18.0.16, path 65001
2019/12/07 02:05:58.210 BGP: 198.18.0.16 rcvd UPDATE wlen 0 attrlen 49 alen 8
2019/12/07 02:05:58.210 BGP: 198.18.0.16 rcvd 172.16.3.0/24 with addpath ID 4 IPv4 unicast
2019/12/07 02:05:58.212 BGP: 198.18.0.4 rcvd UPDATE w/ attr: nexthop 198.18.0.12, origin i, localpref 100, metric 0, originator 198.18.0.12, clusterlist 198.18.0.4, path 65001
2019/12/07 02:05:58.212 BGP: 198.18.0.4 rcvd UPDATE wlen 0 attrlen 49 alen 8
2019/12/07 02:05:58.212 BGP: 198.18.0.4 rcvd 172.16.3.0/24 with addpath ID 3 IPv4 unicast
2019/12/07 02:05:58.212 BGP: 198.18.0.4 rcvd UPDATE w/ attr: nexthop 198.18.0.1, origin i, localpref 100, metric 0, originator 198.18.0.1, clusterlist 198.18.0.4, path 65001
2019/12/07 02:05:58.212 BGP: 198.18.0.4 rcvd UPDATE wlen 0 attrlen 49 alen 8
2019/12/07 02:05:58.212 BGP: 198.18.0.4 rcvd 172.16.3.0/24 with addpath ID 4 IPv4 unicast
2019/12/07 02:05:58.262 BGP: group_announce_route_walkcb: afi=IPv4, safi=unicast, p=172.16.3.0/24
2019/12/07 02:05:58.262 BGP: subgroup_process_announce_selected: p=172.16.3.0/24, selected=0x55c215f69ae0
2019/12/07 02:05:58.262 BGP: group_announce_route_walkcb: afi=IPv4, safi=unicast, p=172.16.3.0/24
2019/12/07 02:05:58.262 BGP: subgroup_process_announce_selected: p=172.16.3.0/24, selected=0x55c215f69ae0
2019/12/07 02:05:58.263 BGP: u2:s2 send UPDATE w/ attr: nexthop 0.0.0.0, origin i, localpref 100, originator 198.18.0.1, clusterlist 198.18.0.4, path 65001
2019/12/07 02:05:58.263 BGP: u2:s2 send UPDATE 172.16.3.0/24 IPv4 unicast
2019/12/07 02:05:58.263 BGP: u2:s2 send UPDATE len 52 numpfx 1
2019/12/07 02:05:58.263 BGP: u2:s2 192.168.254.5 send UPDATE w/ nexthop 192.168.254.1

removing network 172.16.3.0/24 remotely

2019/12/07 02:06:46.341 BGP: 198.18.0.4 rcvd UPDATE wlen 16 attrlen 0 alen 0
2019/12/07 02:06:46.341 BGP: 198.18.0.4 rcvd UPDATE about 172.16.3.0/24 with addpath ID 3 IPv4 unicast -- withdrawn
2019/12/07 02:06:46.341 BGP: 198.18.0.4 rcvd UPDATE about 172.16.3.0/24 with addpath ID 4 IPv4 unicast -- withdrawn
2019/12/07 02:06:46.341 BGP: 198.18.0.16 rcvd UPDATE wlen 16 attrlen 0 alen 0
2019/12/07 02:06:46.341 BGP: 198.18.0.16 rcvd UPDATE about 172.16.3.0/24 with addpath ID 3 IPv4 unicast -- withdrawn
2019/12/07 02:06:46.341 BGP: 198.18.0.16 rcvd UPDATE about 172.16.3.0/24 with addpath ID 4 IPv4 unicast -- withdrawn
2019/12/07 02:06:46.391 BGP: group_announce_route_walkcb: afi=IPv4, safi=unicast, p=172.16.3.0/24
2019/12/07 02:06:46.391 BGP: group_announce_route_walkcb: afi=IPv4, safi=unicast, p=172.16.3.0/24
2019/12/07 02:06:46.391 BGP: subgroup_process_announce_selected: p=172.16.3.0/24, selected=0x0
2019/12/07 02:06:46.392 BGP: u2:s2 send UPDATE 172.16.3.0/24 IPv4 unicast -- unreachable
2019/12/07 02:06:46.392 BGP: u2:s2 send UPDATE (withdraw) len 27 numpfx 1

Output of frr.log in FRR 7.2 with debug bgp update on cgw-4-east1: adding network 172.16.3.0/24 remotely

2019/12/07 01:57:58.138 BGP: 198.18.0.16 rcvd UPDATE w/ attr: nexthop 198.18.0.12, origin i, localpref 100, metric 0, originator 198.18.0.12, clusterlist 198.18.0.16, path 65001
2019/12/07 01:57:58.138 BGP: 198.18.0.16 rcvd UPDATE wlen 0 attrlen 49 alen 8
2019/12/07 01:57:58.138 BGP: 198.18.0.16 rcvd 172.16.3.0/24 with addpath ID 4 IPv4 unicast
2019/12/07 01:57:58.138 BGP: 198.18.0.4 rcvd UPDATE w/ attr: nexthop 198.18.0.12, origin i, localpref 100, metric 0, originator 198.18.0.12, clusterlist 198.18.0.4, path 65001
2019/12/07 01:57:58.138 BGP: 198.18.0.4 rcvd UPDATE wlen 0 attrlen 49 alen 8
2019/12/07 01:57:58.138 BGP: 198.18.0.4 rcvd 172.16.3.0/24 with addpath ID 3 IPv4 unicast
2019/12/07 01:57:58.178 BGP: 198.18.0.4 rcvd UPDATE w/ attr: nexthop 198.18.0.1, origin i, localpref 100, metric 0, originator 198.18.0.1, clusterlist 198.18.0.4, path 65001
2019/12/07 01:57:58.178 BGP: 198.18.0.4 rcvd UPDATE wlen 0 attrlen 49 alen 8
2019/12/07 01:57:58.178 BGP: 198.18.0.4 rcvd 172.16.3.0/24 with addpath ID 4 IPv4 unicast...duplicate ignored
2019/12/07 01:57:58.181 BGP: 198.18.0.16 rcvd UPDATE w/ attr: nexthop 198.18.0.1, origin i, localpref 100, metric 0, originator 198.18.0.1, clusterlist 198.18.0.16, path 65001
2019/12/07 01:57:58.181 BGP: 198.18.0.16 rcvd UPDATE wlen 0 attrlen 49 alen 8
2019/12/07 01:57:58.181 BGP: 198.18.0.16 rcvd 172.16.3.0/24 with addpath ID 3 IPv4 unicast
2019/12/07 01:57:58.188 BGP: group_announce_route_walkcb: afi=IPv4, safi=unicast, p=172.16.3.0/24
2019/12/07 01:57:58.188 BGP: subgroup_process_announce_selected: p=172.16.3.0/24, selected=0x55c215f6b920
2019/12/07 01:57:58.188 BGP: group_announce_route_walkcb: afi=IPv4, safi=unicast, p=172.16.3.0/24
2019/12/07 01:57:58.188 BGP: subgroup_process_announce_selected: p=172.16.3.0/24, selected=0x55c215f6b920
2019/12/07 01:57:58.189 BGP: u2:s2 send UPDATE w/ attr: nexthop 0.0.0.0, origin i, localpref 100, originator 198.18.0.1, clusterlist 198.18.0.4, path 65001
2019/12/07 01:57:58.189 BGP: u2:s2 send UPDATE 172.16.3.0/24 IPv4 unicast
2019/12/07 01:57:58.189 BGP: u2:s2 send UPDATE len 52 numpfx 1
2019/12/07 01:57:58.189 BGP: u2:s2 192.168.254.5 send UPDATE w/ nexthop 192.168.254.1

removing network 172.16.3.0/24 remotely

2019/12/07 01:58:25.905 BGP: 198.18.0.4 rcvd UPDATE wlen 8 attrlen 0 alen 0
2019/12/07 01:58:25.905 BGP: 198.18.0.4 rcvd UPDATE about 172.16.3.0/24 with addpath ID 3 IPv4 unicast -- withdrawn
2019/12/07 01:58:25.905 BGP: 198.18.0.16 rcvd UPDATE wlen 8 attrlen 0 alen 0
2019/12/07 01:58:25.905 BGP: 198.18.0.16 rcvd UPDATE about 172.16.3.0/24 with addpath ID 4 IPv4 unicast -- withdrawn
2019/12/07 01:58:25.955 BGP: group_announce_route_walkcb: afi=IPv4, safi=unicast, p=172.16.3.0/24
2019/12/07 01:58:25.956 BGP: subgroup_process_announce_selected: p=172.16.3.0/24, selected=0x55c215f6b920
2019/12/07 01:58:25.956 BGP: group_announce_route_walkcb: afi=IPv4, safi=unicast, p=172.16.3.0/24
2019/12/07 01:58:25.956 BGP: subgroup_process_announce_selected: p=172.16.3.0/24, selected=0x55c215f6b920
2019/12/07 01:58:25.956 BGP: u2:s2 send UPDATE w/ attr: nexthop 0.0.0.0, origin i, localpref 100, originator 198.18.0.1, clusterlist 198.18.0.4, path 65001
2019/12/07 01:58:25.956 BGP: u2:s2 send UPDATE 172.16.3.0/24 IPv4 unicast
2019/12/07 01:58:25.956 BGP: u2:s2 send UPDATE len 52 numpfx 1
2019/12/07 01:58:25.956 BGP: u2:s2 192.168.254.5 send UPDATE w/ nexthop 192.168.254.1

You can see 2 withdraws are missing in FRR 7.2 compared to 6.x. Even with a single route reflector, the same behavior is noticed, just not with as many duplicate routes.

(put "x" in "[ ]" if you already tried following) [X] Did you check if this is a duplicate issue? Possible source of issue: https://github.com/FRRouting/frr/pull/3051 [ ] Did you test it on the latest FRRouting/frr master branch?

To Reproduce Steps to reproduce the behavior:

                                         ctr-1            ctr-2
                                          |                |
                                /====================================\
                              cgw-5-west1(AS 4238440709)              \
                             /                                         \
vpc-3-west1(AS 65001)========                                      cgw-4-east1(AS 4238440709)
                             \                                         /
                              cgw-6-west1(AS 4238440709)              /
                               \=====================================/

One network (AS 4238440709) has 5 bgp routers:

  1. two are route reflectors with different cluster ids (ctr-1 and ctr-2)
  2. two are peering externally with AS 65001 (cgw-5-west1 and cgw-6-west1)
  3. one internal peer (possibly peering externally, but not relevant to issue) (cgw-4-east1)

The other network, vpc-3-west1, has AS 65001 and wants to advertise network 172.16.3.0/24 to AS 4238440709. Observations of routes were taken from cgw-4-east1.

ctr-1 frr.conf

frr defaults traditional
hostname ctr-1
no log monitor
log timestamp precision 3
log file /var/log/frr/frr.log informational
service integrated-vtysh-config
agentx
!
interface eth0
 description SDN INTERFACE
 ip address 198.18.0.4/16
!
router bgp 4238440709
 bgp router-id 198.18.0.4
 bgp log-neighbor-changes
 bgp cluster-id 198.18.0.4
 no bgp default ipv4-unicast
 timers bgp 20 60
 neighbor rrclients peer-group
 neighbor rrclients remote-as 4238440709
 neighbor rrclients advertisement-interval 1
 bgp listen range 198.18.0.0/16 peer-group rrclients
 !
 address-family ipv4 unicast
  neighbor rrclients activate
  neighbor rrclients addpath-tx-all-paths
  neighbor rrclients route-reflector-client
  neighbor rrclients soft-reconfiguration inbound
 exit-address-family
!
line vty
!

ctr-2 frr.conf

frr defaults traditional
hostname ctr-2
no log monitor
log timestamp precision 3
log file /var/log/frr/frr.log informational
service integrated-vtysh-config
agentx
!
interface eth0
 description SDN INTERFACE
 ip address 198.18.0.16/16
!
router bgp 4238440709
 bgp router-id 198.18.0.16
 bgp log-neighbor-changes
 bgp cluster-id 198.18.0.16
 no bgp default ipv4-unicast
 timers bgp 20 60
 neighbor rrclients peer-group
 neighbor rrclients remote-as 4238440709
 neighbor rrclients advertisement-interval 1
 bgp listen range 198.18.0.0/16 peer-group rrclients
 !
 address-family ipv4 unicast
  neighbor rrclients activate
  neighbor rrclients addpath-tx-all-paths
  neighbor rrclients route-reflector-client
  neighbor rrclients soft-reconfiguration inbound
 exit-address-family
!
line vty
!

cgw-5-west1 frr.conf

frr defaults traditional
hostname cgw-5-west1
no log monitor
log timestamp precision 3
log file /var/log/frr/frr.log informational
service integrated-vtysh-config
agentx
!
!
interface eth1
 description SDN INTERFACE
 ip address 198.18.0.1/16
!
interface eth0
 description REMOTE INTERFACE
 ip address 192.168.254.16/24
!
router bgp 4238440709
 bgp router-id 198.18.0.1
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 timers bgp 20 60
 neighbor rtrflctr peer-group
 neighbor rtrflctr remote-as 4238440709
 neighbor rtrflctr advertisement-interval 1
 neighbor 192.168.254.7 remote-as 65001
 neighbor 192.168.254.7 ebgp-multihop 4
 neighbor 192.168.254.7 local-as 394351 no-prepend replace-as
 neighbor 192.168.254.7 password somekindofpassword
 neighbor 198.18.0.4 peer-group rtrflctr
 neighbor 198.18.0.16 peer-group rtrflctr
!
 address-family ipv4 unicast
  neighbor rtrflctr activate
  neighbor rtrflctr next-hop-self
  neighbor rtrflctr soft-reconfiguration inbound
  neighbor 192.168.254.7 activate
  neighbor 192.168.254.7 soft-reconfiguration inbound
 exit-address-family
!
line vty
!

cgw-6-west1 frr.conf

frr defaults traditional
hostname cgw-6-west1
no log monitor
log timestamp precision 3
log file /var/log/frr/frr.log informational
service integrated-vtysh-config
agentx
!
!
interface eth0
 description SDN INTERFACE
 ip address 198.18.0.12/16
!
interface eth1
 description REMOTE INTERFACE
 ip address 192.168.254.28/24
!
router bgp 4238440709
 bgp router-id 198.18.0.12
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 timers bgp 20 60
 neighbor rtrflctr peer-group
 neighbor rtrflctr remote-as 4238440709
 neighbor rtrflctr advertisement-interval 1
 neighbor 192.168.254.6 remote-as 65001
 neighbor 192.168.254.6 ebgp-multihop 4
 neighbor 192.168.254.6 local-as 394351 no-prepend replace-as
 neighbor 192.168.254.6 password somekindofpassword
 neighbor 198.18.0.4 peer-group rtrflctr
 neighbor 198.18.0.16 peer-group rtrflctr
!
 address-family ipv4 unicast
  neighbor rtrflctr activate
  neighbor rtrflctr next-hop-self
  neighbor rtrflctr soft-reconfiguration inbound
  neighbor 192.168.254.6 activate
  neighbor 192.168.254.6 soft-reconfiguration inbound
 exit-address-family
!
line vty
!

cgw-4-east1 frr.conf

frr defaults traditional
hostname cgw-4-east1
no log monitor
log timestamp precision 3
log file /var/log/frr/frr.log debug
debug bgp updates
service integrated-vtysh-config
agentx
!
!
interface eth1
 description SDN INTERFACE
 ip address 198.18.0.7/16
!
interface eth0
 description REMOTE INTERFACE
 ip address 192.168.254.1/24
!
router bgp 4238440709
 bgp router-id 198.18.0.7
 bgp log-neighbor-changes
 no bgp default ipv4-unicast
 timers bgp 20 60
 neighbor rtrflctr peer-group
 neighbor rtrflctr remote-as 4238440709
 neighbor rtrflctr advertisement-interval 1
 neighbor 192.168.254.5 remote-as 65001
 neighbor 192.168.254.5 ebgp-multihop 4
 neighbor 192.168.254.5 local-as 394351 no-prepend replace-as
 neighbor 192.168.254.5 password somekindofpassword
 neighbor 198.18.0.4 peer-group rtrflctr
 neighbor 198.18.0.16 peer-group rtrflctr
!
 address-family ipv4 unicast
  neighbor rtrflctr activate
  neighbor rtrflctr next-hop-self
  neighbor rtrflctr soft-reconfiguration inbound
  neighbor 192.168.254.5 activate
  neighbor 192.168.254.5 soft-reconfiguration inbound
 exit-address-family
!
line vty
!
  1. Setup the above network using FRR 7.2 (only the route reflectors need to be 7.2 for this to fail)
  2. From vpc-3-west-1 advertise network 172.16.3.0/24
  3. On cgw-4-east1, run show ip bgp and see 4 routes added for 172.16.3.0/24 
    Network          Next Hop            Metric LocPrf Weight Path
*> 172.16.2.0/24    192.168.254.5                          0 65001 i
* i172.16.3.0/24    198.18.0.1               0    100      0 65001 i
* i                 198.18.0.12              0    100      0 65001 i
*>i                 198.18.0.1               0    100      0 65001 i
*=i                 198.18.0.12              0    100      0 65001 i
  4. Now remove advertisement from vpc-3-west-1 of 172.16.3.0/24
  5. On cgw-4-east1, run show ip bgp again and see some routes remaining when they should have all been removed. 
    Network          Next Hop            Metric LocPrf Weight Path
*> 172.16.2.0/24    192.168.254.5                          0 65001 i
* i172.16.3.0/24    198.18.0.1               0    100      0 65001 i
*>i                 198.18.0.1               0    100      0 65001 i

    I think the issue can be observed with a less complex setup, but this is what I was working with when I decided to document the issue.

Expected behavior Assuming that the configurations used are correct (and they work with FRR 6.x), the routes of all duplicate routes should be removed in FRR 7.2 using addpath-tx-all-paths with one or more route reflectors.

Versions

ton31337 commented 4 years ago

@Try75 are you able to test with the latest 7.2 build or the master branch? Because this should be fixed by https://github.com/FRRouting/frr/commit/d4a0d83bfde0e0834edcf3d261fc542286c2fec2

Try75 commented 4 years ago

I have tested with FRR-7.2 debian package from https://deb.frrouting.org/. I think I thought https://github.com/FRRouting/frr/commit/d4a0d83bfde0e0834edcf3d261fc542286c2fec2 was included in the 7.2 debian package I tried, but maybe the package was made before that commit. Is there somewhere I can get a latest 7.2 dev package or a nightly build?

ton31337 commented 4 years ago

@Try75 let me double-check if this is a valid issue.

ton31337 commented 4 years ago

Confirmed, it's a bug.

ton31337 commented 4 years ago

The problem is that route reflectors generate TX IDs for 172.16.3.0/24 (let's say 2 and 5 - from two peers) and send an UPDATE message to route reflector client. When 172.16.3.0/24 is removed, route reflectors send WITHDRAW message only for ID: 2. And for ID 5 WITHDRAW message is not sent and the route with ID 5 sits in the RIB at route reflector clients.

ton31337 commented 4 years ago

fixed by https://github.com/FRRouting/frr/pull/5664

nicoturp commented 4 years ago

We've seen also this bug on our infrastructure. I confirm also that's fixed by #5664

ton31337 commented 4 years ago

fixed for 7.2 and master branches (7.3 included as well).