search

FRRouting / frr

The FRRouting Protocol Suite
https://frrouting.org/
Other
3.34k stars 1.25k forks source link

IPv6 recursive static route that resolves to another static with a directly-connected nexthop rejected by kernel #8109

Open sglaus opened 3 years ago

sglaus commented 3 years ago

Describe the bug Using FRR 7.5.

I have an IPv6 static route configured (2::1/128):

!
ipv6 route 2::1/128 dp0s7.102
ipv6 route 2::1/128 dp0s6
!
dut-2# sh ipv route
Codes: K - kernel route, C - connected, S - static, R - RIPng,
       O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table,
       v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

C>* 1::1/128 is directly connected, lo1, 03:33:09
C>* 1::2/128 is directly connected, lo2, 03:33:08
C>* 1::3/128 is directly connected, lo3, 03:33:11
S>* 2::1/128 [1/0] is directly connected, dp0s6, weight 1, 00:22:04
  *                is directly connected, dp0s7.102, weight 1, 00:22:04
C>* 10:1::/64 is directly connected, dp0s10, 03:33:04
S>* 10:2::/32 [1/0] via 192:168:1::2, dp0s6, weight 1, 00:25:08
S>* 10:2::/64 [1/0] via 192:168:1::2, dp0s6, weight 1, 00:25:08
C>* 192:168:1::/64 is directly connected, dp0s6, 00:25:08
C>* 192:168:2::/64 is directly connected, dp0s7.102, 03:03:46
C>* 192:168:3::/64 is directly connected, dp0s7.103, 03:03:46
C>* 192:168:4::/64 is directly connected, dp0s7.104, 03:03:46
C>* 192:168:5::/64 is directly connected, dp0s7.105, 03:03:46
C>* 192:168:6::/64 is directly connected, dp0s7.106, 03:03:46
C>* 192:168:7::/64 is directly connected, dp0s7.107, 03:03:46

The routes to 2::1/128 have been configured with nexthop interfaces. If I now configure another static route, and use 2::1/128 as the nexthop, it is rejected by the kernel:

dut-2# config
dut-2(config)# ipv6 route 51:1:1::/64 2::1
dut-2(config)# exit
dut-2# sh ipv route
Codes: K - kernel route, C - connected, S - static, R - RIPng,
       O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table,
       v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

C>* 1::1/128 is directly connected, lo1, 03:33:09
C>* 1::2/128 is directly connected, lo2, 03:33:08
C>* 1::3/128 is directly connected, lo3, 03:33:11
S>* 2::1/128 [1/0] is directly connected, dp0s6, weight 1, 00:22:04
  *                is directly connected, dp0s7.102, weight 1, 00:22:04
C>* 10:1::/64 is directly connected, dp0s10, 03:33:04
S>* 10:2::/32 [1/0] via 192:168:1::2, dp0s6, weight 1, 00:25:08
S>* 10:2::/64 [1/0] via 192:168:1::2, dp0s6, weight 1, 00:25:08
S>r 51:1:1::/64 [1/0] via 2::1 (recursive), weight 1, 00:00:19
  r                     via 2::1, dp0s6 onlink, weight 1, 00:00:19
  r                     via 2::1, dp0s7.102 onlink, weight 1, 00:00:19
C>* 192:168:1::/64 is directly connected, dp0s6, 00:25:08
C>* 192:168:2::/64 is directly connected, dp0s7.102, 03:03:46
C>* 192:168:3::/64 is directly connected, dp0s7.103, 03:03:46
C>* 192:168:4::/64 is directly connected, dp0s7.104, 03:03:46
C>* 192:168:5::/64 is directly connected, dp0s7.105, 03:03:46
C>* 192:168:6::/64 is directly connected, dp0s7.106, 03:03:46
C>* 192:168:7::/64 is directly connected, dp0s7.107, 03:03:46

...and I see these errors in the log:

Feb 18 16:45:51 dut-2 zebra[2201]: zebra message[ZEBRA_NEXTHOP_REGISTER:0:30] comes from socket [66]
Feb 18 16:45:51 dut-2 zebra[2201]: Read 1 packets from client: static
Feb 18 16:45:51 dut-2 zebra[2201]: rnh_register msg from client static: hdr->length=20, type=nexthop vrf=0
Feb 18 16:45:51 dut-2 zebra[2201]: 0: Add RNH 2::1/128 type Nexthop
Feb 18 16:45:51 dut-2 zebra[2201]: 0:2::1/128: Evaluate RNH, type Nexthop (force)
Feb 18 16:45:51 dut-2 zebra[2201]: 0:2::1/128: NH resolved over route 2::1/128
Feb 18 16:45:51 dut-2 zebra[2201]: 0: Client static registers for RNH 2::1/128 type Nexthop
Feb 18 16:45:51 dut-2 zebra[2201]: zebra message[ZEBRA_ROUTE_ADD:0:61] comes from socket [66]
Feb 18 16:45:51 dut-2 zebra[2201]: Read 1 packets from client: static
Feb 18 16:45:51 dut-2 zebra[2201]: zread_route_add: p=(0:0)51:1:1::/64, msg flags=0x3, flags=0x41
Feb 18 16:45:51 dut-2 zebra[2201]: nexthop_from_zapi: nh=2::1, vrf_id=0
Feb 18 16:45:51 dut-2 zebra[2201]: zread_route_add: nh=2::1 if 0, vrf_id=0
Feb 18 16:45:51 dut-2 zebra[2201]: rib_add_multipath_nhe: (0:0):51:1:1::/64: Inserting route rn 0x55a24bad5ad0, re 0x55a24bac1450 (static) existing (nil)
Feb 18 16:45:51 dut-2 zebra[2201]: default(0:254):51:1:1::/64: Adding route rn 0x55a24bad5ad0, re 0x55a24bac1450 (static)
Feb 18 16:45:51 dut-2 zebra[2201]: netlink_nexthop_msg_encode: ID (634): 2::1, via dp0s6(17) vrf default(0)
Feb 18 16:45:51 dut-2 zebra[2201]: netlink_nexthop_msg_encode: RTM_NEWNEXTHOP, id=634
Feb 18 16:45:51 dut-2 zebra[2201]: netlink_nexthop_msg_encode: ID (635): 2::1, via dp0s7.102(26) vrf default(0)
Feb 18 16:45:51 dut-2 zebra[2201]: netlink_nexthop_msg_encode: RTM_NEWNEXTHOP, id=635
Feb 18 16:45:51 dut-2 zebra[2201]: _netlink_nexthop_build_group: ID (633): group 634/635
Feb 18 16:45:51 dut-2 zebra[2201]: netlink_nexthop_msg_encode: RTM_NEWNEXTHOP, id=633
Feb 18 16:45:51 dut-2 zebra[2201]: netlink_route_multipath_msg_encode: RTM_NEWROUTE 51:1:1::/64 vrf 0(254)
Feb 18 16:45:51 dut-2 zebra[2201]: netlink_route_multipath_msg_encode: 51:1:1::/64 nhg_id is 633
Feb 18 16:45:51 dut-2 zebra[2201]: nl_batch_send: netlink-dp (NS 0), batch size=236, msg cnt=4
Feb 18 16:45:51 dut-2 zebra[2201]: Extended Error: Nexthop has invalid gateway or device mismatch
Feb 18 16:45:51 dut-2 zebra[2201]: [EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWNEXTHOP(104), seq=603, pid=3818530223
Feb 18 16:45:51 dut-2 zebra[2201]: nl_batch_read_resp: netlink error message seq=603
Feb 18 16:45:51 dut-2 zebra[2201]: Extended Error: Invalid nexthop id
Feb 18 16:45:51 dut-2 zebra[2201]: [EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWNEXTHOP(104), seq=604, pid=3818530223
Feb 18 16:45:51 dut-2 zebra[2201]: nl_batch_read_resp: netlink error message seq=604
Feb 18 16:45:51 dut-2 zebra[2201]: Extended Error: Nexthop id does not exist
Feb 18 16:45:51 dut-2 zebra[2201]: [EC 4043309093] netlink-dp (NS 0) error: Invalid argument, type=RTM_NEWROUTE(24), seq=605, pid=3818530223
Feb 18 16:45:51 dut-2 zebra[2201]: nl_batch_read_resp: netlink error message seq=605
Feb 18 16:45:51 dut-2 zebra[2201]: [EC 4043309074] Failed to install Nexthop ID (635) into the kernel
Feb 18 16:45:51 dut-2 zebra[2201]: [EC 4043309074] Failed to install Nexthop ID (633) into the kernel
Feb 18 16:45:51 dut-2 zebra[2201]: Notifying Owner: static about prefix 51:1:1::/64(254) 0 vrf: 0
Feb 18 16:45:51 dut-2 zebra[2201]: default(0:254):51:1:1::/64: Route install failed
Feb 18 16:45:51 dut-2 staticd[2331]: route_notify_owner: Route 51:1:1::/64 failed to install for table: 254

If I re-configure the 2::1/128 static, and configure nexthop addresses instead of interfaces, then I can add the 51:1:1::/64 static without any problem, so this issue seems related to the use of nexthop interfaces with recursive routes.

Interestingly, if I configure a recursive static with a mix of nexthop interface and nexthop addresses, the static is no longer flagged as rejected (even though it is). In the example below, 51:1:1::/64 is configured with two nexthops - 2::1/128 and 2::2/128.

dut-2# sh ipv6 route
Codes: K - kernel route, C - connected, S - static, R - RIPng,
       O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table,
       v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

C>* 1::1/128 is directly connected, lo1, 04:11:08
C>* 1::2/128 is directly connected, lo2, 04:11:07
C>* 1::3/128 is directly connected, lo3, 04:11:10
S>* 2::1/128 [1/0] is directly connected, dp0s6, weight 1, 00:13:44
  *                is directly connected, dp0s7.102, weight 1, 00:13:44
S>* 2::2/128 [1/0] via 192:168:3::2, dp0s7.103, weight 1, 01:00:03
  *                via 192:168:4::2, dp0s7.104, weight 1, 01:00:03
  *                via 192:168:5::2, dp0s7.105, weight 1, 01:00:03
C>* 10:1::/64 is directly connected, dp0s10, 04:11:03
S>* 10:2::/32 [1/0] via 192:168:1::2, dp0s6, weight 1, 01:03:07
S>* 10:2::/64 [1/0] via 192:168:1::2, dp0s6, weight 1, 01:03:07
S>  51:1:1::/64 [1/0] via 2::1 (recursive), weight 1, 00:00:19
  *                     via 2::1, dp0s6 onlink, weight 1, 00:00:19
  *                     via 2::1, dp0s7.102 onlink, weight 1, 00:00:19
                      via 2::2 (recursive), weight 1, 00:00:19
  *                     via 192:168:3::2, dp0s7.103, weight 1, 00:00:19
  *                     via 192:168:4::2, dp0s7.104, weight 1, 00:00:19
  *                     via 192:168:5::2, dp0s7.105, weight 1, 00:00:19
C>* 192:168:1::/64 is directly connected, dp0s6, 01:03:07
C>* 192:168:2::/64 is directly connected, dp0s7.102, 03:41:45
C>* 192:168:3::/64 is directly connected, dp0s7.103, 03:41:45
C>* 192:168:4::/64 is directly connected, dp0s7.104, 03:41:45
C>* 192:168:5::/64 is directly connected, dp0s7.105, 03:41:45
C>* 192:168:6::/64 is directly connected, dp0s7.106, 03:41:45
C>* 192:168:7::/64 is directly connected, dp0s7.107, 03:41:45

The RIB has flagged 51:1:1::/64 as FIB route, even though the route has not been installed in the kernel:

vyatta@dut-2:~$ ip -6 route
::1 dev lo proto kernel metric 256 pref medium
1::1 dev lo1 proto kernel metric 256 pref medium
1::2 dev lo2 proto kernel metric 256 pref medium
1::3 dev lo3 proto kernel metric 256 pref medium
2::1 nhid 545 proto static metric 20
    nexthop dev dp0s6 weight 1
    nexthop dev dp0s7.102 weight 1 pref medium
2::2 nhid 548 proto static metric 20
    nexthop via 192:168:3::2 dev dp0s7.103 weight 1
    nexthop via 192:168:4::2 dev dp0s7.104 weight 1
    nexthop via 192:168:5::2 dev dp0s7.105 weight 1 pref medium
10:1::/64 dev dp0s10 proto kernel metric 256 pref medium
10:2::/64 nhid 523 via 192:168:1::2 dev dp0s6 proto static metric 20 pref medium
10:2::/32 nhid 523 via 192:168:1::2 dev dp0s6 proto static metric 20 pref medium
192:168:1::/64 dev dp0s6 proto kernel metric 256 pref medium
192:168:2::/64 dev dp0s7.102 proto kernel metric 256 pref medium
192:168:3::/64 dev dp0s7.103 proto kernel metric 256 pref medium
192:168:4::/64 dev dp0s7.104 proto kernel metric 256 pref medium
192:168:5::/64 dev dp0s7.105 proto kernel metric 256 pref medium
192:168:6::/64 dev dp0s7.106 proto kernel metric 256 pref medium
192:168:7::/64 dev dp0s7.107 proto kernel metric 256 pref medium

[x] Did you check if this is a duplicate issue? [x] Did you test it on the latest FRRouting/frr master branch?

idryzhov commented 3 years ago

For someone looking into this in the future – I spent some time on this and I think it's a kernel issue. I was testing on kernel 5.4.

Here is a list of commands to reproduce all the messages FRR sends to the kernel in this case:

ip link add dummy0 type dummy
ip link set dummy0 up
ip link add dummy1 type dummy
ip link set dummy1 up

ip nexthop add id 100 dev dummy0
ip nexthop add id 101 dev dummy1
ip nexthop add id 102 group 100/101
ip route add 1.1.1.1/32 nhid 102

ip nexthop add id 103 via 1.1.1.1 dev dummy0 onlink
ip nexthop add id 104 via 1.1.1.1 dev dummy1 onlink
ip nexthop add id 105 group 103/104
ip route add 2.2.2.2/32 nhid 105

ip -6 nexthop add id 200 dev dummy0
ip -6 nexthop add id 201 dev dummy1
ip nexthop add id 202 group 200/201
ip -6 route add 1::1/128 nhid 202

ip -6 nexthop add id 203 via 1::1 dev dummy0 onlink
ip -6 nexthop add id 204 via 1::1 dev dummy1 onlink  <-- this command fails with "Nexthop has invalid gateway or device mismatch."
ip nexthop add id 205 group 203/204
ip -6 route add 2::2/128 nhid 205

As we can see, when we're trying to add the second onlink IPv6 nexthop, kernel rejects it with "Nexthop has invalid gateway or device mismatch." error. The same set of commands works perfectly in IPv4 case above. By the way, we can successfully add the second IPv6 nexthop without the onlink flag and it works, which is also not correct I suppose.