Route from loopback being invalidated

[ianbarrere]

Hello,

We have a health check which relies on exabgp conditionally advertising an anycast route to FRR which should then forward the route into our network infrastructure. FRR and exabgp are running on the same box. The route itself is an IP from a directly connected loopback interface. FRR considers the route invalid for some reason and thus does not advertise it.

To Reproduce

Loopback configuration from host:

[root@cacp]~# ifconfig lo_bgp
lo_bgp: flags=195<UP,BROADCAST,RUNNING,NOARP>  mtu 1500
        inet 10.255.255.252  netmask 255.255.255.255  broadcast 0.0.0.0
        ether ae:e7:eb:b6:79:d5  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3  bytes 126 (126.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@cacp]~#

FRR config:

Current configuration:
!
frr version 9.0.1
frr defaults traditional
hostname cacp
allow-reserved-ranges
no ipv6 forwarding
!
router bgp 64512
 bgp router-id 10.6.4.47
 bgp log-neighbor-changes
 no bgp network import-check
 neighbor 10.6.4.2 remote-as 64560
 neighbor 10.6.4.3 remote-as 64560
 neighbor 127.0.0.2 remote-as 64512
 neighbor 127.0.0.2 description local-exabgp-daemon
 neighbor 127.0.0.2 passive
 bgp allow-martian-nexthop
 !
 address-family ipv4 unicast
  neighbor 10.6.4.2 prefix-list DEFAULT in
  neighbor 10.6.4.2 prefix-list squid_cast out
  neighbor 10.6.4.2 route-map squid_cast out
  neighbor 10.6.4.3 prefix-list DEFAULT in
  neighbor 10.6.4.3 prefix-list squid_cast out
  neighbor 10.6.4.3 route-map squid_cast out
  neighbor 127.0.0.2 soft-reconfiguration inbound
  neighbor 127.0.0.2 prefix-list squid_cast in
  neighbor 127.0.0.2 prefix-list NONE out
 exit-address-family
exit
!
ip prefix-list squid_cast seq 5 permit 10.255.255.252/32
ip prefix-list DEFAULT seq 5 permit 0.0.0.0/0
ip prefix-list DEFAULT seq 10 deny 0.0.0.0/0 le 32
ip prefix-list NONE seq 5 deny 0.0.0.0/0 le 32
!
route-map squid_cast permit 10
 match ip address prefix-list squid_cast
exit
!
route-map squid_cast deny 99
exit
!
end

The peering with the local exabgp daemon is working fine, and FRR is receiving the route, but it considers it inaccessible/invalid and it's not advertising it:

cacp# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, A - Babel, F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

B   0.0.0.0/0 [20/0] via 10.6.4.2, ens192, weight 1, 00:22:49
                     via 10.6.4.3, ens192, weight 1, 00:22:49
K>* 0.0.0.0/0 [0/100] via 10.6.4.1, ens192, 00:22:52
C>* 10.6.4.0/23 is directly connected, ens192, 00:22:52
C>* 10.255.255.252/32 is directly connected, lo_bgp, 00:22:52
cacp# show ip bgp 10.255.255.252
BGP routing table entry for 10.255.255.252/32, version 0
Paths: (1 available, no best path)
  Not advertised to any peer
  Local
    10.255.255.252 (inaccessible) from 127.0.0.2 (127.0.0.2)
      Origin IGP, localpref 100, invalid, internal
      Last update: Tue Oct  3 14:41:48 2023
cacp#

Expected behavior I would think that FRR should advertise the route as it is originated from a directly connected interface.

Versions

• OS Version: Red Hat Enterprise Linux release 8.8 (Ootpa)
• Kernel: Linux 4.18.0
• FRR Version: 9.0.1 (frr-9.0.1-01.el8.x86_64)

[jlixfeld]

Might redistribute connected in router bgp be helpful?

[cgucker]

the route is in the routing table, so it's not a redistribute issue.

C>* 10.255.255.252/32 is directly connected, lo_bgp, 00:22:52

[ianbarrere]

Might redistribute connected in router bgp be helpful?

We need it conditionally announced though, redistribute connected would advertise it at all times. What we are trying to achieve is a sort of dead man switch, where the route is only advertised when the health check is functional and passing. If the health check conditions fail, if the health check code itself fails, i.e. if anything related to the health check fails we want to withdraw the route, which is why we're doing it through exa.

[ton31337]

Can you show what are you injecting from the ExaBGP side? What you actually have to do is to set a proper next-hop from the ExaBGP side to a valid next-hop from a shared network (most likely 10.6.4.0/23 in your case).

[ianbarrere]

Can you show what are you injecting from the ExaBGP side? What you actually have to do is to set a proper next-hop from the ExaBGP side to a valid next-hop from a shared network (most likely 10.6.4.0/23 in your case).

Hello, ExaBGP is injecting 10.255.255.252/32 with a next-hop of 10.255.255.252. We assumed that that would work since the loopback interface has that same IP. ExaBGP and FRR are peering over another loopback interface with the address 127.0.0.2, and we tried setting that as the next hop as well which also didn't work.

The route is being originated from inside the same device though, so advertising a next-hop of something off the box doesn't really make sense. Can you explain what you mean by "proper" next hop though? Does it need to be an IP on a physical interface or something?

[donaldsharp]

FRR does not allow routes to be recursive through themselves.

[ianbarrere]

That possibility crossed my mind, but then I was wondering why it was also invalid when the next hop was 127.0.0.2.

[cgucker]

That possibility crossed my mind, but then I was wondering why it was also invalid when the next hop was 127.0.0.2.

I haven't tried this, but would 0.0.0.0 be considered a valid next-hop? Many devices will use default as the next hop for covering routes being injected into BGP with underpinning static routes to null0, where covering or more specific routes exist for the actual next-hop forwarding

From a Cisco IOS-XR device: Status codes: s suppressed, d damped, h history, valid, > best i - internal, r RIB-failure, S stale, N Nexthop-discard Origin codes: i - IGP, e - EGP, ? - incomplete Network Next Hop Metric LocPrf Weight Path > X.X.X.0/23 0.0.0.0 0 32768 i

X.X.X.0/23 is being injected via a network statement under BGP, this would be the same thing except its being introduced by exaBGP and then more specifically forwarded to the local loopback interface.

[donaldsharp]

I say give it a try. What could go wrong?

[ton31337]

Setting 0.0.0.0 as the next-hop would mean (self), but I can't say 100%. In any case, this shouldn't work, because from the ExaBGP side, you are setting the wrong next-hop anyway (127.0.0.2). Can you show the config of ExaBGP?

[ianbarrere]

Using 0.0.0.0 indeed not not work, I just tried it. New error, but roughly the same:

cacp# show ip bgp 10.255.255.252
BGP routing table entry for 10.255.255.252/32, version 0
Paths: (1 available, no best path)
  Not advertised to any peer
  Local
    0.0.0.0 (inaccessible) from 127.0.0.2 (127.0.0.2)
      Origin IGP, localpref 100, invalid, internal
      Last update: Thu Oct  5 10:47:32 2023
cacp#

ExaBGP config is very minimal:

process announce-routes {
   run /root/bin/health_check.sh;
   encoder json;
}

neighbor 127.0.0.1 {
        description "Local BGP daemon";
        router-id 127.0.0.2;
        local-address 127.0.0.2;
        local-as 64512;
        peer-as 64512;
        family {
           ipv4 unicast;
        }
        api {
           processes [announce-routes];
        }
}

The health check code just runs announce route 10.255.255.252/32 next-hop 0.0.0.0 (or whatever other next hop we have been testing with) when things are in good shape.

[ton31337]

announce route 10.255.255.252/32 next-hop self also does not work? I've just tested quickly:

neighbor 127.0.0.2 {
  description "ToR";
  router-id 192.168.10.21;
  local-address 127.0.0.1;
  local-as 65001;
  peer-as 65001;
  hold-time 90;
  family {
    ipv4 unicast;
  }
  static {
    route 10.255.255.252/32 next-hop self;
  }
}

and

allow-reserved-ranges
!
router bgp 65001
 no bgp ebgp-requires-policy
 no bgp network import-check
 neighbor 127.0.0.1 remote-as internal
exit
!

Result is:

donatas-pc# show ip bgp 10.255.255.252/32
BGP routing table entry for 10.255.255.252/32, version 3
Paths: (1 available, best #1, table default)
  Not advertised to any peer
  Local
    127.0.0.1 (metric 100) from 127.0.0.1 (192.168.10.21)
      Origin IGP, localpref 100, valid, internal, best (First path received)
      Last update: Thu Oct  5 18:20:16 2023
donatas-pc# 

[ianbarrere]

It looks like this was actually because of an unexpected setup on our side. The next hop address of the route was not explicitly in the interface table on the linux host (although ExaBGP was successfully operating on it). After adding it the route shows up properly. I'm assuming that FRR does some verification to ensure that the next-hop is reachable via one of the interfaces listed in ifconfig or the routing table?

[ton31337]

For sure, the next hop address should be in the RIB.