IPv6 BGP / Zebra - Failed to enqueue dataplane install

FRR: 8.3.1 OS: Debian 11.5 Kernel: 5.10.140

Describe the bug

[ x ] Did you check if this is a duplicate issue?
[ ] Did you test it on the latest FRRouting/frr master branch?

I think there is a bug in the way Zebra logs errors when it can't insert IPv6 prefixes.

'zebra[3869179]: [SWQK6-6JY63][EC 4043309074] 0:254:2c0f:f6d0:27::/48: Failed to enqueue dataplane install'

I think the '0:254' part before the route is wrong.

To Reproduce

Not easy to reproduce. We believe we received about 55k IPv6 prefixes from a peer at a peering point. This caused the router to partially fail. After noticing errors in the frr log, zebra could not insert routes, we restarted frr and routing returned to normal.

Additional context

Somebody noticed something similar recently, also contains the '0:254' https://github.com/FRRouting/frr/issues/10199

The log file does not give us enough information to trace the problem in detail, but maybe fixing the logging is a good start.

In what way is 0:254 wrong in your opinion? What leads you to this conclusion?

Hi, all 55k routes had the same ~prepend, and if you prepend that you cant find the routes in WHOIS. If you take away the ~prepend, you can find many of the routes in WHOIS.

CPU load went crazy at that point, but we don't really know what went wrong.

the [0:254:... is telling us that the route was destined for the default vrf and for table 254 ( the default routing table ). If you need more data about what went wrong I would turn on debug zebra kernel and debug zebra dplane and see where it leads us. There are a bunch of places that the data plane can tell us that a route install failed and we cannot decode the problem from what you have given us.

At the point that log event gets generated, do you know the reason for it failing? That would be useful info to have even in non debug mode.

About the '0:254' thing. Seems like it could be much more user friendly if you printed: [ vrf 0 | table 254 ]

and not make it look like a v6 prefix

Just leaving this here for somebody who does a search for the FRR error - investigate setting your linux kernel ipv6 route cache sysctl variables. I suspect kernel maybe went crazy trying to do v6 route cache garbage collection, but not sure.

Here is a reproduce for this issue. This is a server with four unnumbered peers on four VLAN interfaces (two on each physical interface).

systemctl restart frr # start fresh
ip link set down dev eno1 # down one interface, two peers die
# wait some seconds
ip link set up dev eno1 # up the interface again and traffic on eno1 gets blackholed

Versions:

Ubuntu 22.04.1
Kernel 5.15.0-57-generic
FRR 8.4.1-0~ubuntu22.04.1

Interfaces:

2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether e4:43:4b:63:ce:64 brd ff:ff:ff:ff:ff:ff
    altname enp25s0f0
    inet6 fe80::e643:4bff:fe63:ce64/64 scope link 
       valid_lft forever preferred_lft forever
4: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether e4:43:4b:63:ce:66 brd ff:ff:ff:ff:ff:ff
    altname enp25s0f1
    inet6 fe80::e643:4bff:fe63:ce66/64 scope link 
       valid_lft forever preferred_lft forever
5: eno4: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether e4:43:4b:63:ce:85 brd ff:ff:ff:ff:ff:ff
    altname enp1s0f1
6: vlan.709@eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether e4:43:4b:63:ce:66 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::e643:4bff:fe63:ce66/64 scope link 
       valid_lft forever preferred_lft forever
7: vlan.759@eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether e4:43:4b:63:ce:66 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::e643:4bff:fe63:ce66/64 scope link 
       valid_lft forever preferred_lft forever
8: vlan.708@eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether e4:43:4b:63:ce:64 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::e643:4bff:fe63:ce64/64 scope link 
       valid_lft forever preferred_lft forever
9: vlan.758@eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether e4:43:4b:63:ce:64 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::e643:4bff:fe63:ce64/64 scope link 
       valid_lft forever preferred_lft forever

Config

frr version 8.4.1
frr defaults traditional
hostname server
log file /var/log/frr/frr.log
log syslog informational
no ip forwarding
no ipv6 forwarding
service integrated-vtysh-config
!
debug zebra kernel
debug zebra dplane
debug zebra dplane dpdk
debug zebra nexthop
!
router bgp 65201
 bgp router-id 5.5.5.5
 bgp log-neighbor-changes
 neighbor vlan.708 interface remote-as external
 neighbor vlan.708 bfd
 neighbor vlan.709 interface remote-as external
 neighbor vlan.709 bfd
 neighbor vlan.758 interface remote-as external
 neighbor vlan.758 bfd
 neighbor vlan.759 interface remote-as external
 neighbor vlan.759 bfd
 !
 address-family ipv4 unicast
  redistribute connected
  neighbor vlan.708 soft-reconfiguration inbound
  neighbor vlan.708 distribute-list in-v4 in
  neighbor vlan.708 distribute-list out-v4 out
  neighbor vlan.709 soft-reconfiguration inbound
  neighbor vlan.709 distribute-list in-v4 in
  neighbor vlan.709 distribute-list out-v4 out
  neighbor vlan.758 soft-reconfiguration inbound
  neighbor vlan.758 distribute-list in-v4 in
  neighbor vlan.758 distribute-list out-v4 out
  neighbor vlan.759 soft-reconfiguration inbound
  neighbor vlan.759 distribute-list in-v4 in
  neighbor vlan.759 distribute-list out-v4 out
 exit-address-family
exit
!
access-list out-v4 seq 1 permit 1.1.1.1/32
access-list in-v4 seq 1 permit 0.0.0.0/0
!
end

Log from above reproduce (from interface down and forward)

2023/01/20 10:48:50 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=88, seq=0, pid=0
2023/01/20 10:48:50 ZEBRA: [KKAC1-JMWTB] Rx RTM_NEWNEIGH family ipv6 IF vlan.759(7) vrf default(0) IP fe80::e643:4bff:fe63:7b98 MAC e4:43:4b:63:7b:98 state 0x10 flags 0x80 ext_flags 0x0
2023/01/20 10:48:50 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=88, seq=0, pid=0
2023/01/20 10:48:50 ZEBRA: [KKAC1-JMWTB] Rx RTM_NEWNEIGH family ipv6 IF vlan.759(7) vrf default(0) IP fe80::e643:4bff:fe63:7b98 MAC e4:43:4b:63:7b:98 state 0x2 flags 0x80 ext_flags 0x0
2023/01/20 10:48:50 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=88, seq=0, pid=0
2023/01/20 10:48:50 ZEBRA: [KKAC1-JMWTB] Rx RTM_NEWNEIGH family ipv6 IF vlan.709(6) vrf default(0) IP fe80::e643:4bff:fe0b:42da MAC e4:43:4b:0b:42:da state 0x4 flags 0x80 ext_flags 0x0
2023/01/20 10:48:55 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=88, seq=0, pid=0
2023/01/20 10:48:55 ZEBRA: [KKAC1-JMWTB] Rx RTM_NEWNEIGH family ipv6 IF vlan.709(6) vrf default(0) IP fe80::e643:4bff:fe0b:42da MAC e4:43:4b:0b:42:da state 0x10 flags 0x80 ext_flags 0x0
2023/01/20 10:48:55 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=88, seq=0, pid=0
2023/01/20 10:48:55 ZEBRA: [KKAC1-JMWTB] Rx RTM_NEWNEIGH family ipv6 IF vlan.709(6) vrf default(0) IP fe80::e643:4bff:fe0b:42da MAC e4:43:4b:0b:42:da state 0x2 flags 0x80 ext_flags 0x0
2023/01/20 10:49:00 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=88, seq=0, pid=0
2023/01/20 10:49:00 ZEBRA: [KKAC1-JMWTB] Rx RTM_NEWNEIGH family ipv6 IF vlan.758(9) vrf default(0) IP fe80::e643:4bff:fe63:7b96 MAC e4:43:4b:63:7b:96 state 0x4 flags 0x80 ext_flags 0x0
2023/01/20 10:49:01 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWLINK(16), len=1408, seq=0, pid=0
2023/01/20 10:49:01 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_DELADDR(21), len=72, seq=0, pid=0
2023/01/20 10:49:01 ZEBRA: [RGWF1-EHXT1] netlink_interface_addr_dplane: RTM_DELADDR nsid 0 ifindex 2 flags 0x80:
2023/01/20 10:49:01 ZEBRA: [ME3M2-X6YT9]   IFA_ADDRESS   fe80::e643:4bff:fe63:ce64/64
2023/01/20 10:49:01 ZEBRA: [P2VPT-508WP]   IFA_CACHEINFO pref -1, valid -1
2023/01/20 10:49:01 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWLINK(16), len=1408, seq=0, pid=0
2023/01/20 10:49:01 ZEBRA: [W6BZR-YZPAB] RTM_NEWLINK update for eno1(2) sl_type 0 master 0 flags 0x1002
2023/01/20 10:49:01 ZEBRA: [MR3ZF-ATDBY] Intf eno1(2) has gone DOWN
2023/01/20 10:49:01 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELROUTE(25), len=116, seq=0, pid=0
2023/01/20 10:49:01 ZEBRA: [SKNFJ-G938V] RTM_DELROUTE ipv6 unicast proto kernel NS 0
2023/01/20 10:49:01 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELROUTE(25), len=116, seq=0, pid=0
2023/01/20 10:49:01 ZEBRA: [SKNFJ-G938V] RTM_DELROUTE ipv6 local proto kernel NS 0
2023/01/20 10:49:01 ZEBRA: [J3J81-V75NW] Route rtm_type: local(2) intentionally ignoring
2023/01/20 10:49:01 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELROUTE(25), len=116, seq=0, pid=0
2023/01/20 10:49:01 ZEBRA: [SKNFJ-G938V] RTM_DELROUTE ipv6 multicast proto kernel NS 0
2023/01/20 10:49:01 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELNEIGH(29), len=88, seq=0, pid=0
2023/01/20 10:49:02 ZEBRA: [K8FXY-V65ZJ] Intf dplane ctx 0x7f9e78026f30, op INTF_ADDR_DEL, ifindex (2), result QUEUED
2023/01/20 10:49:02 ZEBRA: [MZPZA-W042K] zebra_if_addr_update_ctx: INTF_ADDR_DEL: ifindex eno1(2), addr fe80::e643:4bff:fe63:ce64/64
2023/01/20 10:49:02 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELNEIGH(29), len=88, seq=0, pid=0
2023/01/20 10:49:02 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELNEIGH(29), len=88, seq=0, pid=0
2023/01/20 10:49:02 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELADDR(21), len=72, seq=0, pid=0
2023/01/20 10:49:02 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWLINK(16), len=1400, seq=0, pid=0
2023/01/20 10:49:02 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_DELADDR(21), len=72, seq=0, pid=0
2023/01/20 10:49:02 ZEBRA: [RGWF1-EHXT1] netlink_interface_addr_dplane: RTM_DELADDR nsid 0 ifindex 9 flags 0x80:
2023/01/20 10:49:02 ZEBRA: [ME3M2-X6YT9]   IFA_ADDRESS   fe80::e643:4bff:fe63:ce64/64
2023/01/20 10:49:02 ZEBRA: [P2VPT-508WP]   IFA_CACHEINFO pref -1, valid -1
2023/01/20 10:49:02 BGP: [PXVXG-TFNNT] %ADJCHANGE: neighbor vlan.758(router2) in vrf default Down Waiting for NHT
2023/01/20 10:49:02 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWLINK(16), len=1400, seq=0, pid=0
2023/01/20 10:49:02 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_DELADDR(21), len=72, seq=0, pid=0
2023/01/20 10:49:02 ZEBRA: [RGWF1-EHXT1] netlink_interface_addr_dplane: RTM_DELADDR nsid 0 ifindex 8 flags 0x80:
2023/01/20 10:49:02 ZEBRA: [ME3M2-X6YT9]   IFA_ADDRESS   fe80::e643:4bff:fe63:ce64/64
2023/01/20 10:49:02 ZEBRA: [P2VPT-508WP]   IFA_CACHEINFO pref -1, valid -1
2023/01/20 10:49:02 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWLINK(16), len=1400, seq=0, pid=0
2023/01/20 10:49:02 ZEBRA: [W6BZR-YZPAB] RTM_NEWLINK update for vlan.758(9) sl_type 0 master 0 flags 0x1002
2023/01/20 10:49:02 ZEBRA: [MR3ZF-ATDBY] Intf vlan.758(9) has gone DOWN
2023/01/20 10:49:02 ZEBRA: [HCFTH-1NSTK] netlink_talk: netlink-cmd (NS 0) type RTM_DELNEIGH(29), len=56 seq=18 flags 0x401
2023/01/20 10:49:02 ZEBRA: [MQ5AP-2S1F5] netlink-cmd (NS 0) error: No such file or directory, type=RTM_DELNEIGH(29), seq=18, pid=2309414657
2023/01/20 10:49:02 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELNEIGH(29), len=84, seq=0, pid=0
2023/01/20 10:49:02 ZEBRA: [TCTVW-JD3B6]     Neighbor Entry Received is a 5549 entry, finished
2023/01/20 10:49:02 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELROUTE(25), len=116, seq=0, pid=0
2023/01/20 10:49:02 ZEBRA: [SKNFJ-G938V] RTM_DELROUTE ipv6 unicast proto kernel NS 0
2023/01/20 10:49:02 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELROUTE(25), len=116, seq=0, pid=0
2023/01/20 10:49:02 ZEBRA: [SKNFJ-G938V] RTM_DELROUTE ipv6 local proto kernel NS 0
2023/01/20 10:49:02 ZEBRA: [J3J81-V75NW] Route rtm_type: local(2) intentionally ignoring
2023/01/20 10:49:02 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELROUTE(25), len=116, seq=0, pid=0
2023/01/20 10:49:02 ZEBRA: [SKNFJ-G938V] RTM_DELROUTE ipv6 multicast proto kernel NS 0
2023/01/20 10:49:02 ZEBRA: [K8FXY-V65ZJ] Intf dplane ctx 0x7f9e78026f30, op INTF_ADDR_DEL, ifindex (9), result QUEUED
2023/01/20 10:49:02 ZEBRA: [MZPZA-W042K] zebra_if_addr_update_ctx: INTF_ADDR_DEL: ifindex vlan.758(9), addr fe80::e643:4bff:fe63:ce64/64
2023/01/20 10:49:02 ZEBRA: [K8FXY-V65ZJ] Intf dplane ctx 0x7f9e780278a0, op INTF_ADDR_DEL, ifindex (8), result QUEUED
2023/01/20 10:49:02 ZEBRA: [MZPZA-W042K] zebra_if_addr_update_ctx: INTF_ADDR_DEL: ifindex vlan.708(8), addr fe80::e643:4bff:fe63:ce64/64
2023/01/20 10:49:02 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELNEIGH(29), len=88, seq=0, pid=0
2023/01/20 10:49:02 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELNEIGH(29), len=88, seq=0, pid=0
2023/01/20 10:49:02 ZEBRA: [YK42S-VD2K1] Rx RTM_DELNEIGH family ipv6 IF vlan.758(9) vrf default(0) IP fe80::e643:4bff:fe63:7b96
2023/01/20 10:49:02 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELNEIGH(29), len=88, seq=0, pid=0
2023/01/20 10:49:02 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELNEIGH(29), len=88, seq=0, pid=0
2023/01/20 10:49:02 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELNEIGH(29), len=88, seq=0, pid=0
2023/01/20 10:49:02 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELADDR(21), len=72, seq=0, pid=0
2023/01/20 10:49:02 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWLINK(16), len=1400, seq=0, pid=0
2023/01/20 10:49:02 ZEBRA: [W6BZR-YZPAB] RTM_NEWLINK update for vlan.708(8) sl_type 0 master 0 flags 0x1002
2023/01/20 10:49:02 ZEBRA: [MR3ZF-ATDBY] Intf vlan.708(8) has gone DOWN
2023/01/20 10:49:02 ZEBRA: [HCFTH-1NSTK] netlink_talk: netlink-cmd (NS 0) type RTM_DELNEIGH(29), len=56 seq=19 flags 0x401
2023/01/20 10:49:02 ZEBRA: [MQ5AP-2S1F5] netlink-cmd (NS 0) error: No such file or directory, type=RTM_DELNEIGH(29), seq=19, pid=2309414657
2023/01/20 10:49:02 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELNEIGH(29), len=84, seq=0, pid=0
2023/01/20 10:49:02 ZEBRA: [TCTVW-JD3B6]     Neighbor Entry Received is a 5549 entry, finished
2023/01/20 10:49:02 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELROUTE(25), len=116, seq=0, pid=0
2023/01/20 10:49:02 ZEBRA: [SKNFJ-G938V] RTM_DELROUTE ipv6 unicast proto kernel NS 0
2023/01/20 10:49:02 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELROUTE(25), len=116, seq=0, pid=0
2023/01/20 10:49:02 ZEBRA: [SKNFJ-G938V] RTM_DELROUTE ipv6 local proto kernel NS 0
2023/01/20 10:49:02 ZEBRA: [J3J81-V75NW] Route rtm_type: local(2) intentionally ignoring
2023/01/20 10:49:02 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELROUTE(25), len=116, seq=0, pid=0
2023/01/20 10:49:02 ZEBRA: [SKNFJ-G938V] RTM_DELROUTE ipv6 multicast proto kernel NS 0
2023/01/20 10:49:02 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELNEIGH(29), len=88, seq=0, pid=0
2023/01/20 10:49:02 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELNEIGH(29), len=88, seq=0, pid=0
2023/01/20 10:49:02 ZEBRA: [YK42S-VD2K1] Rx RTM_DELNEIGH family ipv6 IF vlan.708(8) vrf default(0) IP fe80::e643:4bff:fe0b:42d8
2023/01/20 10:49:02 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELNEIGH(29), len=88, seq=0, pid=0
2023/01/20 10:49:02 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELNEIGH(29), len=88, seq=0, pid=0
2023/01/20 10:49:02 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELNEIGH(29), len=88, seq=0, pid=0
2023/01/20 10:49:02 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELADDR(21), len=72, seq=0, pid=0
2023/01/20 10:49:02 BGP: [PXVXG-TFNNT] %ADJCHANGE: neighbor vlan.708(router1) in vrf default Down Waiting for NHT
2023/01/20 10:49:02 ZEBRA: [SWQK6-6JY63][EC 4043309074] 0:254:fe80::/64: Failed to enqueue dataplane install
2023/01/20 10:49:02 ZEBRA: [VC7VN-KSJDY] zebra_nhg_nhe2grp_internal: Nexthop ID (328) not valid, not appending to dataplane install group
2023/01/20 10:49:02 ZEBRA: [VC7VN-KSJDY] zebra_nhg_nhe2grp_internal: Nexthop ID (329) not valid, not appending to dataplane install group
2023/01/20 10:49:02 ZEBRA: [VNMVB-91G3G] _netlink_nexthop_build_group: ID (327): group 315/317
2023/01/20 10:49:02 ZEBRA: [R43C6-KYHWT] netlink_nexthop_msg_encode: RTM_NEWNEXTHOP, id=327
2023/01/20 10:49:02 ZEBRA: [HYEHE-CQZ9G] nl_batch_send: netlink-dp (NS 0), batch size=52, msg cnt=1
2023/01/20 10:49:02 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELNEXTHOP(105), len=60, seq=2272591315, pid=1
2023/01/20 10:49:02 ZEBRA: [KBCV5-6W9G6] RTM_DELNEXTHOP ID (313)  NS 0
2023/01/20 10:49:02 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELNEXTHOP(105), len=52, seq=2272591316, pid=1
2023/01/20 10:49:02 ZEBRA: [KBCV5-6W9G6] RTM_DELNEXTHOP ID (321)  NS 0
2023/01/20 10:49:02 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (313[314/315/316/317]) that we are still using for a route, sending it back down
2023/01/20 10:49:02 ZEBRA: [VC7VN-KSJDY] zebra_nhg_nhe2grp_internal: Nexthop ID (314) not valid, not appending to dataplane install group
2023/01/20 10:49:02 ZEBRA: [VC7VN-KSJDY] zebra_nhg_nhe2grp_internal: Nexthop ID (316) not valid, not appending to dataplane install group
2023/01/20 10:49:02 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (321[314/315]) that we are still using for a route, sending it back down
2023/01/20 10:49:02 ZEBRA: [VC7VN-KSJDY] zebra_nhg_nhe2grp_internal: Nexthop ID (314) not valid, not appending to dataplane install group
2023/01/20 10:49:02 ZEBRA: [VNMVB-91G3G] _netlink_nexthop_build_group: ID (313): group 315/317
2023/01/20 10:49:02 ZEBRA: [R43C6-KYHWT] netlink_nexthop_msg_encode: RTM_NEWNEXTHOP, id=313
2023/01/20 10:49:02 ZEBRA: [VNMVB-91G3G] _netlink_nexthop_build_group: ID (321): group 315
2023/01/20 10:49:02 ZEBRA: [R43C6-KYHWT] netlink_nexthop_msg_encode: RTM_NEWNEXTHOP, id=321
2023/01/20 10:49:02 ZEBRA: [HYEHE-CQZ9G] nl_batch_send: netlink-dp (NS 0), batch size=96, msg cnt=2
2023/01/20 10:49:02 ZEBRA: [VNMVB-91G3G] _netlink_nexthop_build_group: ID (333): group 315/317
2023/01/20 10:49:02 ZEBRA: [R43C6-KYHWT] netlink_nexthop_msg_encode: RTM_NEWNEXTHOP, id=333
2023/01/20 10:49:02 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 0.0.0.0/0 vrf 0(254)
2023/01/20 10:49:02 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 0.0.0.0/0 nhg_id is 333
2023/01/20 10:49:02 ZEBRA: [HYEHE-CQZ9G] nl_batch_send: netlink-dp (NS 0), batch size=104, msg cnt=2
2023/01/20 10:49:08 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=88, seq=0, pid=0
2023/01/20 10:49:08 ZEBRA: [KKAC1-JMWTB] Rx RTM_NEWNEIGH family ipv6 IF vlan.759(7) vrf default(0) IP fe80::e643:4bff:fe63:7b98 MAC e4:43:4b:63:7b:98 state 0x4 flags 0x80 ext_flags 0x0
2023/01/20 10:49:13 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=88, seq=0, pid=0
2023/01/20 10:49:13 ZEBRA: [KKAC1-JMWTB] Rx RTM_NEWNEIGH family ipv6 IF vlan.759(7) vrf default(0) IP fe80::e643:4bff:fe63:7b98 MAC e4:43:4b:63:7b:98 state 0x10 flags 0x80 ext_flags 0x0
2023/01/20 10:49:13 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=88, seq=0, pid=0
2023/01/20 10:49:13 ZEBRA: [KKAC1-JMWTB] Rx RTM_NEWNEIGH family ipv6 IF vlan.759(7) vrf default(0) IP fe80::e643:4bff:fe63:7b98 MAC e4:43:4b:63:7b:98 state 0x2 flags 0x80 ext_flags 0x0
2023/01/20 10:49:32 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELROUTE(25), len=116, seq=0, pid=0
2023/01/20 10:49:32 ZEBRA: [SKNFJ-G938V] RTM_DELROUTE ipv6 unicast proto router advertisement NS 0
2023/01/20 10:49:32 ZEBRA: [K2A4T-TS83H] RTM_DELROUTE ::/0 vrf default(0) table_id: 254 metric: 1024 Admin Distance: 0
2023/01/20 10:49:32 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELROUTE(25), len=116, seq=0, pid=0
2023/01/20 10:49:32 ZEBRA: [SKNFJ-G938V] RTM_DELROUTE ipv6 unicast proto router advertisement NS 0
2023/01/20 10:49:32 ZEBRA: [K2A4T-TS83H] RTM_DELROUTE ::/0 vrf default(0) table_id: 254 metric: 1024 Admin Distance: 0
2023/01/20 10:49:33 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWLINK(16), len=1408, seq=0, pid=0
2023/01/20 10:49:33 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWLINK(16), len=1400, seq=0, pid=0
2023/01/20 10:49:33 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWLINK(16), len=1400, seq=0, pid=0
2023/01/20 10:49:33 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWLINK(16), len=1408, seq=0, pid=0
2023/01/20 10:49:33 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWLINK(16), len=1400, seq=0, pid=0
2023/01/20 10:49:33 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWLINK(16), len=1408, seq=0, pid=0
2023/01/20 10:49:33 ZEBRA: [W6BZR-YZPAB] RTM_NEWLINK update for eno1(2) sl_type 0 master 0 flags 0x11003
2023/01/20 10:49:33 ZEBRA: [MR3ZF-ATDBY] Intf eno1(2) has gone DOWN
2023/01/20 10:49:33 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWLINK(16), len=1400, seq=0, pid=0
2023/01/20 10:49:33 ZEBRA: [W6BZR-YZPAB] RTM_NEWLINK update for vlan.708(8) sl_type 0 master 0 flags 0x11003
2023/01/20 10:49:33 ZEBRA: [MR3ZF-ATDBY] Intf vlan.708(8) has gone DOWN
2023/01/20 10:49:33 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWLINK(16), len=1400, seq=0, pid=0
2023/01/20 10:49:33 ZEBRA: [W6BZR-YZPAB] RTM_NEWLINK update for vlan.758(9) sl_type 0 master 0 flags 0x11003
2023/01/20 10:49:33 ZEBRA: [MR3ZF-ATDBY] Intf vlan.758(9) has gone DOWN
2023/01/20 10:49:33 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=116, seq=0, pid=0
2023/01/20 10:49:33 ZEBRA: [SKNFJ-G938V] RTM_NEWROUTE ipv6 multicast proto kernel NS 0
2023/01/20 10:49:33 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=116, seq=0, pid=0
2023/01/20 10:49:33 ZEBRA: [SKNFJ-G938V] RTM_NEWROUTE ipv6 unicast proto kernel NS 0
2023/01/20 10:49:33 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWLINK(16), len=1400, seq=0, pid=0
2023/01/20 10:49:33 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWLINK(16), len=1408, seq=0, pid=0
2023/01/20 10:49:33 ZEBRA: [W6BZR-YZPAB] RTM_NEWLINK update for eno1(2) sl_type 0 master 0 flags 0x11043
2023/01/20 10:49:33 ZEBRA: [N7FN2-J93A7] Intf eno1(2) has come UP
2023/01/20 10:49:33 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=116, seq=0, pid=0
2023/01/20 10:49:33 ZEBRA: [SKNFJ-G938V] RTM_NEWROUTE ipv6 multicast proto kernel NS 0
2023/01/20 10:49:33 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=116, seq=0, pid=0
2023/01/20 10:49:33 ZEBRA: [SKNFJ-G938V] RTM_NEWROUTE ipv6 unicast proto kernel NS 0
2023/01/20 10:49:33 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWLINK(16), len=1400, seq=0, pid=0
2023/01/20 10:49:33 ZEBRA: [W6BZR-YZPAB] RTM_NEWLINK update for vlan.708(8) sl_type 0 master 0 flags 0x11043
2023/01/20 10:49:33 ZEBRA: [N7FN2-J93A7] Intf vlan.708(8) has come UP
2023/01/20 10:49:33 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=116, seq=0, pid=0
2023/01/20 10:49:33 ZEBRA: [SKNFJ-G938V] RTM_NEWROUTE ipv6 multicast proto kernel NS 0
2023/01/20 10:49:33 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=116, seq=0, pid=0
2023/01/20 10:49:33 ZEBRA: [SKNFJ-G938V] RTM_NEWROUTE ipv6 unicast proto kernel NS 0
2023/01/20 10:49:33 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWLINK(16), len=1400, seq=0, pid=0
2023/01/20 10:49:33 ZEBRA: [W6BZR-YZPAB] RTM_NEWLINK update for vlan.758(9) sl_type 0 master 0 flags 0x11043
2023/01/20 10:49:33 ZEBRA: [N7FN2-J93A7] Intf vlan.758(9) has come UP
2023/01/20 10:49:34 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWADDR(20), len=72, seq=0, pid=0
2023/01/20 10:49:34 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=116, seq=0, pid=0
2023/01/20 10:49:34 ZEBRA: [SKNFJ-G938V] RTM_NEWROUTE ipv6 local proto kernel NS 0
2023/01/20 10:49:34 ZEBRA: [J3J81-V75NW] Route rtm_type: local(2) intentionally ignoring
2023/01/20 10:49:34 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWADDR(20), len=72, seq=0, pid=0
2023/01/20 10:49:34 ZEBRA: [RGWF1-EHXT1] netlink_interface_addr_dplane: RTM_NEWADDR nsid 0 ifindex 2 flags 0x80:
2023/01/20 10:49:34 ZEBRA: [ME3M2-X6YT9]   IFA_ADDRESS   fe80::e643:4bff:fe63:ce64/64
2023/01/20 10:49:34 ZEBRA: [P2VPT-508WP]   IFA_CACHEINFO pref -1, valid -1
2023/01/20 10:49:34 ZEBRA: [K8FXY-V65ZJ] Intf dplane ctx 0x7f9e78026f30, op INTF_ADDR_ADD, ifindex (2), result QUEUED
2023/01/20 10:49:34 ZEBRA: [MZPZA-W042K] zebra_if_addr_update_ctx: INTF_ADDR_ADD: ifindex eno1(2), addr fe80::e643:4bff:fe63:ce64/64
2023/01/20 10:49:34 ZEBRA: [YXCJP-0WZWV] netlink_nexthop_msg_encode: ID (312): directly connected, eno1(2) vrf default(0)  
2023/01/20 10:49:34 ZEBRA: [R43C6-KYHWT] netlink_nexthop_msg_encode: RTM_NEWNEXTHOP, id=312
2023/01/20 10:49:34 ZEBRA: [HYEHE-CQZ9G] nl_batch_send: netlink-dp (NS 0), batch size=40, msg cnt=1
2023/01/20 10:49:34 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWADDR(20), len=72, seq=0, pid=0
2023/01/20 10:49:34 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=116, seq=0, pid=0
2023/01/20 10:49:34 ZEBRA: [SKNFJ-G938V] RTM_NEWROUTE ipv6 local proto kernel NS 0
2023/01/20 10:49:34 ZEBRA: [J3J81-V75NW] Route rtm_type: local(2) intentionally ignoring
2023/01/20 10:49:34 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWADDR(20), len=72, seq=0, pid=0
2023/01/20 10:49:34 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=116, seq=0, pid=0
2023/01/20 10:49:34 ZEBRA: [SKNFJ-G938V] RTM_NEWROUTE ipv6 local proto kernel NS 0
2023/01/20 10:49:34 ZEBRA: [J3J81-V75NW] Route rtm_type: local(2) intentionally ignoring
2023/01/20 10:49:34 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWADDR(20), len=72, seq=0, pid=0
2023/01/20 10:49:34 ZEBRA: [RGWF1-EHXT1] netlink_interface_addr_dplane: RTM_NEWADDR nsid 0 ifindex 9 flags 0x80:
2023/01/20 10:49:34 ZEBRA: [ME3M2-X6YT9]   IFA_ADDRESS   fe80::e643:4bff:fe63:ce64/64
2023/01/20 10:49:34 ZEBRA: [P2VPT-508WP]   IFA_CACHEINFO pref -1, valid -1
2023/01/20 10:49:34 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWADDR(20), len=72, seq=0, pid=0
2023/01/20 10:49:34 ZEBRA: [RGWF1-EHXT1] netlink_interface_addr_dplane: RTM_NEWADDR nsid 0 ifindex 8 flags 0x80:
2023/01/20 10:49:34 ZEBRA: [ME3M2-X6YT9]   IFA_ADDRESS   fe80::e643:4bff:fe63:ce64/64
2023/01/20 10:49:34 ZEBRA: [P2VPT-508WP]   IFA_CACHEINFO pref -1, valid -1
2023/01/20 10:49:34 ZEBRA: [K8FXY-V65ZJ] Intf dplane ctx 0x7f9e78026f30, op INTF_ADDR_ADD, ifindex (9), result QUEUED
2023/01/20 10:49:34 ZEBRA: [MZPZA-W042K] zebra_if_addr_update_ctx: INTF_ADDR_ADD: ifindex vlan.758(9), addr fe80::e643:4bff:fe63:ce64/64
2023/01/20 10:49:34 ZEBRA: [K8FXY-V65ZJ] Intf dplane ctx 0x7f9e780278a0, op INTF_ADDR_ADD, ifindex (8), result QUEUED
2023/01/20 10:49:34 ZEBRA: [MZPZA-W042K] zebra_if_addr_update_ctx: INTF_ADDR_ADD: ifindex vlan.708(8), addr fe80::e643:4bff:fe63:ce64/64
2023/01/20 10:49:35 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=88, seq=0, pid=0
2023/01/20 10:49:35 ZEBRA: [KKAC1-JMWTB] Rx RTM_NEWNEIGH family ipv6 IF vlan.708(8) vrf default(0) IP fe80::e643:4bff:fe0b:42d8 MAC e4:43:4b:0b:42:d8 state 0x2 flags 0x80 ext_flags 0x0
2023/01/20 10:49:35 BGP: [J7484-2SYXF][EC 33554465] vlan.708 [FSM] Failure handling event BGP_Start in state Idle, prior events BGP_Stop, Clearing_Completed, fd -1
2023/01/20 10:49:35 ZEBRA: [HCFTH-1NSTK] netlink_talk: netlink-cmd (NS 0) type RTM_DELNEIGH(29), len=56 seq=20 flags 0x401
2023/01/20 10:49:35 ZEBRA: [MQ5AP-2S1F5] netlink-cmd (NS 0) error: No such file or directory, type=RTM_DELNEIGH(29), seq=20, pid=2309414657
2023/01/20 10:49:35 ZEBRA: [HCFTH-1NSTK] netlink_talk: netlink-cmd (NS 0) type RTM_NEWNEIGH(28), len=56 seq=21 flags 0x401
2023/01/20 10:49:35 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=88, seq=0, pid=0
2023/01/20 10:49:35 ZEBRA: [KKAC1-JMWTB] Rx RTM_NEWNEIGH family ipv6 IF vlan.758(9) vrf default(0) IP fe80::e643:4bff:fe63:7b96 MAC e4:43:4b:63:7b:96 state 0x8 flags 0x80 ext_flags 0x0
2023/01/20 10:49:35 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=176, seq=2272591324, pid=1
2023/01/20 10:49:35 ZEBRA: [SKNFJ-G938V] RTM_NEWROUTE ipv6 unicast proto router advertisement NS 0
2023/01/20 10:49:35 ZEBRA: [K2A4T-TS83H] RTM_NEWROUTE ::/0 vrf default(0) table_id: 254 metric: 1024 Admin Distance: 0
2023/01/20 10:49:35 BGP: [YZN21-6KS42][EC 33554465] vlan.758 [FSM] Ignoring event BGP_Start in state Connect, prior events BGP_Start, BGP_Start, fd 25
2023/01/20 10:49:35 BGP: [HZN6M-XRM1G] %NOTIFICATION: sent to neighbor vlan.758 6/7 (Cease/Connection Collision Resolution) 0 bytes 
2023/01/20 10:49:35 BGP: [HTQD2-0R1WR][EC 33554451] bgp_process_packet: BGP OPEN receipt failed for peer: vlan.758
2023/01/20 10:49:35 BGP: [N9HHH-F8H1M] %ADJCHANGE: neighbor vlan.758(router2) in vrf default Up
2023/01/20 10:49:35 ZEBRA: [YXCJP-0WZWV] netlink_nexthop_msg_encode: ID (316): fe80::e643:4bff:fe63:7b96, via vlan.758(9) vrf default(0)  
2023/01/20 10:49:35 ZEBRA: [R43C6-KYHWT] netlink_nexthop_msg_encode: RTM_NEWNEXTHOP, id=316
2023/01/20 10:49:35 ZEBRA: [VNMVB-91G3G] _netlink_nexthop_build_group: ID (342): group 315/316/317
2023/01/20 10:49:35 ZEBRA: [R43C6-KYHWT] netlink_nexthop_msg_encode: RTM_NEWNEXTHOP, id=342
2023/01/20 10:49:35 ZEBRA: [HYEHE-CQZ9G] nl_batch_send: netlink-dp (NS 0), batch size=120, msg cnt=2
2023/01/20 10:49:38 ZEBRA: [HCFTH-1NSTK] netlink_talk: netlink-cmd (NS 0) type RTM_DELNEIGH(29), len=56 seq=22 flags 0x401
2023/01/20 10:49:38 ZEBRA: [MQ5AP-2S1F5] netlink-cmd (NS 0) error: No such file or directory, type=RTM_DELNEIGH(29), seq=22, pid=2309414657
2023/01/20 10:49:38 ZEBRA: [HCFTH-1NSTK] netlink_talk: netlink-cmd (NS 0) type RTM_NEWNEIGH(28), len=56 seq=23 flags 0x401
2023/01/20 10:49:38 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=204, seq=2272591325, pid=1
2023/01/20 10:49:38 ZEBRA: [SKNFJ-G938V] RTM_NEWROUTE ipv6 unicast proto router advertisement NS 0
2023/01/20 10:49:38 ZEBRA: [K2A4T-TS83H] RTM_NEWROUTE ::/0 vrf default(0) table_id: 254 metric: 1024 Admin Distance: 0
2023/01/20 10:49:38 BGP: [N9HHH-F8H1M] %ADJCHANGE: neighbor vlan.708(router1) in vrf default Up
2023/01/20 10:49:38 ZEBRA: [YXCJP-0WZWV] netlink_nexthop_msg_encode: ID (314): fe80::e643:4bff:fe0b:42d8, via vlan.708(8) vrf default(0)  
2023/01/20 10:49:38 ZEBRA: [R43C6-KYHWT] netlink_nexthop_msg_encode: RTM_NEWNEXTHOP, id=314
2023/01/20 10:49:38 ZEBRA: [HYEHE-CQZ9G] nl_batch_send: netlink-dp (NS 0), batch size=60, msg cnt=1
2023/01/20 10:49:38 BGP: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from vlan.758 in vrf default
2023/01/20 10:49:38 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 0.0.0.0/0 vrf 0(254)
2023/01/20 10:49:38 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 0.0.0.0/0 nhg_id is 342
2023/01/20 10:49:38 ZEBRA: [HYEHE-CQZ9G] nl_batch_send: netlink-dp (NS 0), batch size=52, msg cnt=1
2023/01/20 10:49:40 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=88, seq=0, pid=0
2023/01/20 10:49:40 ZEBRA: [KKAC1-JMWTB] Rx RTM_NEWNEIGH family ipv6 IF vlan.758(9) vrf default(0) IP fe80::e643:4bff:fe63:7b96 MAC e4:43:4b:63:7b:96 state 0x2 flags 0x80 ext_flags 0x0
2023/01/20 10:49:41 BGP: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from vlan.708 in vrf default
2023/01/20 10:49:41 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 0.0.0.0/0 vrf 0(254)
2023/01/20 10:49:41 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 0.0.0.0/0 nhg_id is 313
2023/01/20 10:49:41 ZEBRA: [HYEHE-CQZ9G] nl_batch_send: netlink-dp (NS 0), batch size=52, msg cnt=1
2023/01/20 10:49:43 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=88, seq=0, pid=0
2023/01/20 10:49:43 ZEBRA: [KKAC1-JMWTB] Rx RTM_NEWNEIGH family ipv6 IF vlan.759(7) vrf default(0) IP fe80::e643:4bff:fe63:7b98 MAC e4:43:4b:63:7b:98 state 0x4 flags 0x80 ext_flags 0x0
2023/01/20 10:49:48 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=88, seq=0, pid=0
2023/01/20 10:49:48 ZEBRA: [KKAC1-JMWTB] Rx RTM_NEWNEIGH family ipv6 IF vlan.759(7) vrf default(0) IP fe80::e643:4bff:fe63:7b98 MAC e4:43:4b:63:7b:98 state 0x10 flags 0x80 ext_flags 0x0
2023/01/20 10:49:48 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=88, seq=0, pid=0
2023/01/20 10:49:48 ZEBRA: [KKAC1-JMWTB] Rx RTM_NEWNEIGH family ipv6 IF vlan.759(7) vrf default(0) IP fe80::e643:4bff:fe63:7b98 MAC e4:43:4b:63:7b:98 state 0x2 flags 0x80 ext_flags 0x0
2023/01/20 10:50:11 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=88, seq=0, pid=0
2023/01/20 10:50:11 ZEBRA: [KKAC1-JMWTB] Rx RTM_NEWNEIGH family ipv6 IF vlan.758(9) vrf default(0) IP fe80::e643:4bff:fe63:7b96 MAC e4:43:4b:63:7b:96 state 0x4 flags 0x80 ext_flags 0x0

Netplan config:

network:
  version: 2
  ethernets:
      eno1:
          dhcp4: false
      eno2:
          dhcp4: false
  vlans:
      vlan.708:
          id: 708
          link: eno1
      vlan.709:
          id: 709
          link: eno2
      vlan.758:
          id: 758
          link: eno1
      vlan.759:
          id: 759
          link: eno2

Same issue in 8.4.2-0~ubuntu22.04.1 with above reproduce

Downgrading to 8.1-1ubuntu1.3 (from Ubuntu main repo) makes it work again, was not able to test another version because 8.3 etc is not in frr-8 repo installable

Having the same issue on 8.4.2 flapping the interface repeatedly eventually triggers the problem

from our investigations it seems that zebra and the kernel get out of sync on the next hop groups. zebra references a next hop that no longer exists. fixing/re-creating the nexthop group and clearing the sessions seems to fix the problem without a full restart... (still investigating it further):

ecmp_nh=$(vtysh -c "show ipv6 route ::/0 json" | jq -e '."::/0"[0].installedNexthopGroupId')

# retrieve the dependent next hops for the ecmp nh
IFS=',' read -r nh_1 nh_2 < <(vtysh -c "show nexthop-group rib json" | jq -r ".\"${ecmp_nh}\" | .\"depends\" | @csv")

# install nexthop group in the kernel with info we retrieved above
ip next repl id "${ecmp_nh}" group "${nh_1}"/"${nh_2}" proto zebra

# force a rib update in FRR, this should have a check ensuring there's multipe sessions up
#vtysh -c 'clear bgp ipv6 unicast frr_if0'

with some further testing problem can be reproduced 100%: to reproduce:

ecmp next hop ipv6 route
interface flaps for <160 in our case...

observed behaviour:

if 1 (or both) link flap for <~160 sec:
- when link goes down:
  - zebra keeps dual nh in group id
  - kernel removes the downed link from the group id, but keeps group id with single member
  - next hop installed in fib is the direct next hop of the remaining up interface (not the group one)
- when link comes back up
  - zebra still has original dual nh in group id
  - kernel still has original ecmp nh id with single id member
  - next hop installed in fib is original ecmp nh id with single member in kernel but dual in zebra
if both links flap at the same time, kernel/zebra updates the original ecmp nh group with the same original nh ids for both links. everything works as expected
if both link flap but offset in timing <160: new ecmp nh group id is created with correct slaves
if both (or single) links stay down for >~180 sec, upon coming up new ecmp nh group id is created with correct members, everything works as expected
- original ecmp next hop id is removed from both kernel and zebra
- new ecmp next hop group id is created with original direct next hop ids

expected behavior (spot checking some previous frr versions/kernel:

link goes down: both that direct nexthop as well as ecmp should be eliminated by both the kernel and zebra
link comes back up: new direct and ecmp next hop should be created (new ecmp nh group to include the old functioning direct nh and newly created direct one)

I'm now able to reproduce the issue. Reproduced on stable/8.4 branch (but I believe same issue in master) with Ubuntu 22.04. Verified with IPv6 at this time. Key issues to reproduce this: 1) Route needs to be learned over ECMP path 2) Route uses IPv6 link-local for next-hop 3) FRR needs to use nexthop groups with the kernel (You need to see the nhid in the ip -6 route display.

The above description to reproduce the issue by @tobias-urdin does produce the log message reported, but not the issue.

Here is the setup I'm using:

+--------------+                   +---------------+
|      R1      |                   |      R2       |
|              |                   |               |
|       enp1s0 +-------------------+ enp1s0        |
|              |                   |        enp2s0 +-----
|              |                   |               |
|       enp7s0 +-------------------+ enp8s0        |
|              |                   |               |
+--------------+                   +---------------+

Configurations:

R1

Current configuration:
!
frr version 8.4.2
frr defaults datacenter
hostname bgp-r1
log file /var/log/frr/frr.log
log syslog informational
no ip forwarding
no ipv6 forwarding
zebra dplane limit 2000
service integrated-vtysh-config
!
debug zebra kernel
debug zebra dplane
debug zebra dplane dpdk
debug zebra nexthop
!
interface enp1s0
 ipv6 nd ra-interval 10
 no ipv6 nd suppress-ra
exit
!
interface enp7s0
 ipv6 nd ra-interval 10
 no ipv6 nd suppress-ra
exit
!
router bgp 65201
 bgp router-id 5.5.5.5
 no bgp default ipv4-unicast
 no bgp default show-hostname
 no bgp default show-nexthop-hostname
 no bgp deterministic-med
 coalesce-time 1000
 bgp graceful-restart stalepath-time 15
 bgp graceful-restart
 bgp graceful-restart preserve-fw-state
 bgp bestpath as-path multipath-relax
 neighbor TOR peer-group
 neighbor TOR remote-as external
 neighbor TOR timers 3 9
 neighbor TOR timers connect 5
 neighbor TOR capability extended-nexthop
 neighbor enp1s0 interface peer-group TOR
 neighbor enp7s0 interface peer-group TOR
 !
 address-family ipv4 unicast
  redistribute kernel
  neighbor TOR activate
  neighbor TOR soft-reconfiguration inbound
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute kernel
  neighbor TOR activate
  neighbor TOR soft-reconfiguration inbound
 exit-address-family
exit
!
end

R2

Building configuration...

Current configuration:
!
frr version 8.4.2
frr defaults datacenter
hostname bgp-r2
log file /var/log/frr/frr.log
log syslog informational
no ip forwarding
no ipv6 forwarding
service integrated-vtysh-config
!
debug zebra kernel
debug zebra dplane
debug zebra dplane dpdk
debug zebra nexthop
!
ipv6 route ::/0 2000:1000::10
ipv6 route 2001:1111:1::/48 2000:1000::10
ipv6 route 2002:2222:2::/48 2000:1000::10
ipv6 route 2003:3333:3::/48 2000:1000::10
ipv6 route 2004:4444:4::/48 2000:1000::10
ipv6 route 2005:5555:5::/48 2000:1000::10
ipv6 route 2006:6666:6::/48 2000:1000::10
ipv6 route 2007:7777:7::/48 2000:1000::10
ipv6 route 2008:8888:8::/48 2000:1000::10
ipv6 route 2009:9999:9::/48 2000:1000::10
ipv6 route 200a:aaaa:a::/48 2000:1000::10
ipv6 route 200b:bbbb:b::/48 2000:1000::10
ipv6 route 200c:cccc:c::/48 2000:1000::10
ipv6 route 200d:dddd:d::/48 2000:1000::10
ipv6 route 200e:eeee:e::/48 2000:1000::10
ipv6 route 200f:ffff:f::/48 2000:1000::10
!
interface enp1s0
 description link 1 to R1
exit
!
interface enp2s0
 description Link to someplace else
 ip address 192.168.99.1/24
 ipv6 address 2000:1000::1/64
 ipv6 nd ra-interval 10
 no ipv6 nd suppress-ra
exit
!
interface enp8s0
 description link 2 to R1
exit
!
router bgp 65202
 bgp router-id 6.6.6.6
 no bgp default ipv4-unicast
 no bgp default show-hostname
 no bgp default show-nexthop-hostname
 no bgp deterministic-med
 coalesce-time 1000
 bgp graceful-restart stalepath-time 15
 bgp graceful-restart
 bgp graceful-restart preserve-fw-state
 bgp bestpath as-path multipath-relax
 neighbor TOR peer-group
 neighbor TOR remote-as external
 neighbor TOR timers 3 9
 neighbor TOR timers connect 5
 neighbor TOR capability extended-nexthop
 neighbor enp1s0 interface peer-group TOR
 neighbor enp8s0 interface peer-group TOR
 !
 address-family ipv4 unicast
  network 192.168.99.0/24
  redistribute kernel
  redistribute static
  neighbor TOR activate
  neighbor TOR soft-reconfiguration inbound
 exit-address-family
 !
 address-family ipv6 unicast
  redistribute kernel
  redistribute static
  neighbor TOR activate
  neighbor TOR soft-reconfiguration inbound
 exit-address-family
exit
!
end

Netplan config

R1

(enp2s0 is used for accessing the box, not for the test)

network:
    ethernets:
       enp2s0:
          dhcp4: true
          dhcp-identifier: mac
          dhcp4-overrides:
             use-routes: false
          dhcp6: false
          accept-ra: false
       enp1s0:
          dhcp4: false
          dhcp6: false
          accept-ra: false
       enp7s0:
          dhcp4: false
          dhcp6: false
          accept-ra: false
    version: 2

R2

(enp2s0 is used here for accessing router, not for test)

network:
    ethernets:
       enp3s0:
          dhcp4: true
          dhcp-identifier: mac
          dhcp4-overrides:
             use-routes: false
          dhcp6: false
          accept-ra: false
       enp1s0:
          dhcp4: false
          dhcp6: false
          accept-ra: false
       enp2s0: {}
       enp8s0:
          dhcp4: false
          dhcp6: false
          accept-ra: false
    version: 2

Issue Reproduction

The issue is reproduced on R1. R2 is just used to simulate the routes. First make sure that the routes are using nexthop groups: On R1:

root@bgp-r1:~/frr# ip -6 route show default
default nhid 33 proto bgp metric 20 pref medium
    nexthop via fe80::5054:ff:fe4b:d00d dev enp1s0 weight 1 
    nexthop via fe80::5054:ff:fe13:76ca dev enp7s0 weight 1

It should show the nhid in the route to cause the problem.

In this example we see kernel using nexthop group 33 and FRR can be verified with vtysh -c "show ipv6 route ::/0 json" to use the same nexthop group ("installedNexthopGroupId":33,) Looking at the details with vtysh -c "show nexthop-group rib 33" we can see the individual nexthop groups from FRR:

root@bgp-r1:~/frr# vtysh -c "show nexthop-group rib 33"
ID: 33 (zebra)
     RefCnt: 17
     Uptime: 00:25:39
     VRF: default
     Valid, Installed
     Depends: (29) (34)
           via fe80::5054:ff:fe13:76ca, enp7s0 (vrf default), weight 1
           via fe80::5054:ff:fe4b:d00d, enp1s0 (vrf default), weight 1

and kernel (with ip next get id 33):

root@bgp-r1:~/frr# ip next get id 33
id 33 group 29/34 proto zebra

--> Everything is ok at the start: We have the ecmp nexthop group 33 with the indivdual nexthops 29 & 34.

Now shutdown one of the links on R1 and bring it back up about 10 secs later:

root@bgp-r1:~# ip link set down dev enp7s0
root@bgp-r1:~# sleep 10
root@bgp-r1:~# ip link set up dev enp7s0

Checking FRR:

root@bgp-r1:~/frr# vtysh -c "show ipv6 route ::/0 json" | grep nexthopGroupId
      "nexthopGroupId":33,
root@bgp-r1:~/frr# vtysh -c "show nexthop-group rib 33"
ID: 33 (zebra)
     RefCnt: 17
     Uptime: 00:02:58
     VRF: default
     Valid, Installed
     Depends: (29) (34)
           via fe80::5054:ff:fe13:76ca, enp7s0 (vrf default), weight 1
           via fe80::5054:ff:fe4b:d00d, enp1s0 (vrf default), weight 1

--> Still 33 with 29//34 in FRR

Checking Kernel:

root@bgp-r1:~/frr# ip -6 route show default
default nhid 33 proto bgp metric 20 pref medium
    nexthop via fe80::5054:ff:fe4b:d00d dev enp1s0 weight 1 
root@bgp-r1:~/frr# ip next get id 33
id 33 group 29 proto zebra

--> Kernel has group 33 with only 29 - Nexthop group 34 (for the one on enp7s0) is gone.

As a result of this the ECMP path now only uses one link and if that link flaps, then we have no connectivity anymore.

Below is some log from FRR when this happened.

note the

2023/03/06 16:39:48 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (33[29/34]) that we are still using for a route, sending it back down

and the

2023/03/06 16:39:48 ZEBRA: [VC7VN-KSJDY] zebra_nhg_nhe2grp_internal: Nexthop ID (34) not valid, not appending to dataplane install group

in the log

2023/03/06 16:39:48 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELADDR(21), len=72, seq=0, pid=0
2023/03/06 16:39:48 BGP: [PXVXG-TFNNT] %ADJCHANGE: neighbor enp7s0(bgp-r2) in vrf default Down Waiting for NHT
2023/03/06 16:39:48 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELROUTE(25), len=148, seq=5074942, pid=1
2023/03/06 16:39:48 ZEBRA: [SKNFJ-G938V] RTM_DELROUTE ipv6 unicast proto BGP NS 0
2023/03/06 16:39:48 ZEBRA: [K2A4T-TS83H] RTM_DELROUTE 200c:cccc:c::/48 vrf default(0) table_id: 254 metric: 20 Admin Distance: 0
2023/03/06 16:39:48 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELROUTE(25), len=148, seq=5074943, pid=1
2023/03/06 16:39:48 ZEBRA: [SKNFJ-G938V] RTM_DELROUTE ipv6 unicast proto BGP NS 0
2023/03/06 16:39:48 ZEBRA: [K2A4T-TS83H] RTM_DELROUTE 2002:2222:2::/48 vrf default(0) table_id: 254 metric: 20 Admin Distance: 0
2023/03/06 16:39:48 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELROUTE(25), len=148, seq=5074944, pid=1
2023/03/06 16:39:48 ZEBRA: [SKNFJ-G938V] RTM_DELROUTE ipv6 unicast proto BGP NS 0
2023/03/06 16:39:48 ZEBRA: [K2A4T-TS83H] RTM_DELROUTE 2008:8888:8::/48 vrf default(0) table_id: 254 metric: 20 Admin Distance: 0
2023/03/06 16:39:48 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELROUTE(25), len=148, seq=5074945, pid=1
2023/03/06 16:39:48 ZEBRA: [SKNFJ-G938V] RTM_DELROUTE ipv6 unicast proto BGP NS 0
2023/03/06 16:39:48 ZEBRA: [K2A4T-TS83H] RTM_DELROUTE 200d:dddd:d::/48 vrf default(0) table_id: 254 metric: 20 Admin Distance: 0
2023/03/06 16:39:48 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELROUTE(25), len=148, seq=5074946, pid=1
2023/03/06 16:39:48 ZEBRA: [SKNFJ-G938V] RTM_DELROUTE ipv6 unicast proto BGP NS 0
2023/03/06 16:39:48 ZEBRA: [K2A4T-TS83H] RTM_DELROUTE 200e:eeee:e::/48 vrf default(0) table_id: 254 metric: 20 Admin Distance: 0
2023/03/06 16:39:48 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELROUTE(25), len=92, seq=5074947, pid=1
2023/03/06 16:39:48 ZEBRA: [SKNFJ-G938V] RTM_DELROUTE ipv4 unicast proto BGP NS 0
2023/03/06 16:39:48 ZEBRA: [K2A4T-TS83H] RTM_DELROUTE 192.168.99.0/24 vrf default(0) table_id: 254 metric: 20 Admin Distance: 0
2023/03/06 16:39:48 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELROUTE(25), len=148, seq=5074948, pid=1
2023/03/06 16:39:48 ZEBRA: [SKNFJ-G938V] RTM_DELROUTE ipv6 unicast proto BGP NS 0
2023/03/06 16:39:48 ZEBRA: [K2A4T-TS83H] RTM_DELROUTE 200a:aaaa:a::/48 vrf default(0) table_id: 254 metric: 20 Admin Distance: 0
2023/03/06 16:39:48 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELROUTE(25), len=148, seq=5074949, pid=1
2023/03/06 16:39:48 ZEBRA: [SKNFJ-G938V] RTM_DELROUTE ipv6 unicast proto BGP NS 0
2023/03/06 16:39:48 ZEBRA: [K2A4T-TS83H] RTM_DELROUTE 200f:ffff:f::/48 vrf default(0) table_id: 254 metric: 20 Admin Distance: 0
2023/03/06 16:39:48 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELROUTE(25), len=148, seq=5074950, pid=1
2023/03/06 16:39:48 ZEBRA: [SKNFJ-G938V] RTM_DELROUTE ipv6 unicast proto BGP NS 0
2023/03/06 16:39:48 ZEBRA: [K2A4T-TS83H] RTM_DELROUTE 2006:6666:6::/48 vrf default(0) table_id: 254 metric: 20 Admin Distance: 0
2023/03/06 16:39:48 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELROUTE(25), len=148, seq=5074951, pid=1
2023/03/06 16:39:48 ZEBRA: [SKNFJ-G938V] RTM_DELROUTE ipv6 unicast proto BGP NS 0
2023/03/06 16:39:48 ZEBRA: [K2A4T-TS83H] RTM_DELROUTE 2005:5555:5::/48 vrf default(0) table_id: 254 metric: 20 Admin Distance: 0
2023/03/06 16:39:48 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELROUTE(25), len=148, seq=5074952, pid=1
2023/03/06 16:39:48 ZEBRA: [SKNFJ-G938V] RTM_DELROUTE ipv6 unicast proto BGP NS 0
2023/03/06 16:39:48 ZEBRA: [K2A4T-TS83H] RTM_DELROUTE 2007:7777:7::/48 vrf default(0) table_id: 254 metric: 20 Admin Distance: 0
2023/03/06 16:39:48 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELROUTE(25), len=148, seq=5074953, pid=1
2023/03/06 16:39:48 ZEBRA: [SKNFJ-G938V] RTM_DELROUTE ipv6 unicast proto BGP NS 0
2023/03/06 16:39:48 ZEBRA: [K2A4T-TS83H] RTM_DELROUTE 200b:bbbb:b::/48 vrf default(0) table_id: 254 metric: 20 Admin Distance: 0
2023/03/06 16:39:48 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELROUTE(25), len=128, seq=5074954, pid=1
2023/03/06 16:39:48 ZEBRA: [SKNFJ-G938V] RTM_DELROUTE ipv6 unicast proto BGP NS 0
2023/03/06 16:39:48 ZEBRA: [K2A4T-TS83H] RTM_DELROUTE ::/0 vrf default(0) table_id: 254 metric: 20 Admin Distance: 0
2023/03/06 16:39:48 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELROUTE(25), len=148, seq=5074955, pid=1
2023/03/06 16:39:48 ZEBRA: [SKNFJ-G938V] RTM_DELROUTE ipv6 unicast proto BGP NS 0
2023/03/06 16:39:48 ZEBRA: [K2A4T-TS83H] RTM_DELROUTE 2009:9999:9::/48 vrf default(0) table_id: 254 metric: 20 Admin Distance: 0
2023/03/06 16:39:48 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELROUTE(25), len=148, seq=5074956, pid=1
2023/03/06 16:39:48 ZEBRA: [SKNFJ-G938V] RTM_DELROUTE ipv6 unicast proto BGP NS 0
2023/03/06 16:39:48 ZEBRA: [K2A4T-TS83H] RTM_DELROUTE 2001:1111:1::/48 vrf default(0) table_id: 254 metric: 20 Admin Distance: 0
2023/03/06 16:39:48 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELROUTE(25), len=148, seq=5074957, pid=1
2023/03/06 16:39:48 ZEBRA: [SKNFJ-G938V] RTM_DELROUTE ipv6 unicast proto BGP NS 0
2023/03/06 16:39:48 ZEBRA: [K2A4T-TS83H] RTM_DELROUTE 2003:3333:3::/48 vrf default(0) table_id: 254 metric: 20 Admin Distance: 0
2023/03/06 16:39:48 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELROUTE(25), len=148, seq=5074959, pid=1
2023/03/06 16:39:48 ZEBRA: [SKNFJ-G938V] RTM_DELROUTE ipv6 unicast proto BGP NS 0
2023/03/06 16:39:48 ZEBRA: [K2A4T-TS83H] RTM_DELROUTE 2004:4444:4::/48 vrf default(0) table_id: 254 metric: 20 Admin Distance: 0
2023/03/06 16:39:48 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_DELNEXTHOP(105), len=52, seq=5074962, pid=1
2023/03/06 16:39:48 ZEBRA: [KBCV5-6W9G6] RTM_DELNEXTHOP ID (33)  NS 0
2023/03/06 16:39:48 ZEBRA: [RG2NH-FTSDH][EC 4043309102] Kernel deleted a nexthop group with ID (33[29/34]) that we are still using for a route, sending it back down
2023/03/06 16:39:48 ZEBRA: [VNMVB-91G3G] _netlink_nexthop_build_group: ID (33): group 29
2023/03/06 16:39:48 ZEBRA: [R43C6-KYHWT] netlink_nexthop_msg_encode: RTM_NEWNEXTHOP, id=33
2023/03/06 16:39:48 ZEBRA: [HYEHE-CQZ9G] nl_batch_send: netlink-dp (NS 0), batch size=44, msg cnt=1
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 200c:cccc:c::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 200c:cccc:c::/48 nhg_id is 33
2023/03/06 16:39:48 ZEBRA: [HYEHE-CQZ9G] nl_batch_send: netlink-dp (NS 0), batch size=64, msg cnt=1
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 2002:2222:2::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 2002:2222:2::/48 nhg_id is 33
2023/03/06 16:39:48 ZEBRA: [HYEHE-CQZ9G] nl_batch_send: netlink-dp (NS 0), batch size=64, msg cnt=1
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 2008:8888:8::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 2008:8888:8::/48 nhg_id is 33
2023/03/06 16:39:48 ZEBRA: [HYEHE-CQZ9G] nl_batch_send: netlink-dp (NS 0), batch size=64, msg cnt=1
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 200d:dddd:d::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 200d:dddd:d::/48 nhg_id is 33
2023/03/06 16:39:48 ZEBRA: [HYEHE-CQZ9G] nl_batch_send: netlink-dp (NS 0), batch size=64, msg cnt=1
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 200e:eeee:e::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 200e:eeee:e::/48 nhg_id is 33
2023/03/06 16:39:48 ZEBRA: [HYEHE-CQZ9G] nl_batch_send: netlink-dp (NS 0), batch size=64, msg cnt=1
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 192.168.99.0/24 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 192.168.99.0/24 nhg_id is 33
2023/03/06 16:39:48 ZEBRA: [HYEHE-CQZ9G] nl_batch_send: netlink-dp (NS 0), batch size=52, msg cnt=1
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 200a:aaaa:a::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 200a:aaaa:a::/48 nhg_id is 33
2023/03/06 16:39:48 ZEBRA: [HYEHE-CQZ9G] nl_batch_send: netlink-dp (NS 0), batch size=64, msg cnt=1
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 200f:ffff:f::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 200f:ffff:f::/48 nhg_id is 33
2023/03/06 16:39:48 ZEBRA: [HYEHE-CQZ9G] nl_batch_send: netlink-dp (NS 0), batch size=64, msg cnt=1
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 2006:6666:6::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 2006:6666:6::/48 nhg_id is 33
2023/03/06 16:39:48 ZEBRA: [HYEHE-CQZ9G] nl_batch_send: netlink-dp (NS 0), batch size=64, msg cnt=1
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 2005:5555:5::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 2005:5555:5::/48 nhg_id is 33
2023/03/06 16:39:48 ZEBRA: [HYEHE-CQZ9G] nl_batch_send: netlink-dp (NS 0), batch size=64, msg cnt=1
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 2007:7777:7::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 2007:7777:7::/48 nhg_id is 33
2023/03/06 16:39:48 ZEBRA: [HYEHE-CQZ9G] nl_batch_send: netlink-dp (NS 0), batch size=64, msg cnt=1
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 200b:bbbb:b::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 200b:bbbb:b::/48 nhg_id is 33
2023/03/06 16:39:48 ZEBRA: [HYEHE-CQZ9G] nl_batch_send: netlink-dp (NS 0), batch size=64, msg cnt=1
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE ::/0 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: ::/0 nhg_id is 33
2023/03/06 16:39:48 ZEBRA: [HYEHE-CQZ9G] nl_batch_send: netlink-dp (NS 0), batch size=64, msg cnt=1
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 2009:9999:9::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 2009:9999:9::/48 nhg_id is 33
2023/03/06 16:39:48 ZEBRA: [HYEHE-CQZ9G] nl_batch_send: netlink-dp (NS 0), batch size=64, msg cnt=1
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 2001:1111:1::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 2001:1111:1::/48 nhg_id is 33
2023/03/06 16:39:48 ZEBRA: [HYEHE-CQZ9G] nl_batch_send: netlink-dp (NS 0), batch size=64, msg cnt=1
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 2003:3333:3::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 2003:3333:3::/48 nhg_id is 33
2023/03/06 16:39:48 ZEBRA: [HYEHE-CQZ9G] nl_batch_send: netlink-dp (NS 0), batch size=64, msg cnt=1
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 2004:4444:4::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 2004:4444:4::/48 nhg_id is 33
2023/03/06 16:39:48 ZEBRA: [HYEHE-CQZ9G] nl_batch_send: netlink-dp (NS 0), batch size=64, msg cnt=1
2023/03/06 16:39:48 ZEBRA: [VC7VN-KSJDY] zebra_nhg_nhe2grp_internal: Nexthop ID (34) not valid, not appending to dataplane install group
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 192.168.99.0/24 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 192.168.99.0/24 nhg_id is 29
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_DELROUTE ::/0 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE ::/0 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: ::/0 nhg_id is 29
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_DELROUTE 2001:1111:1::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 2001:1111:1::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 2001:1111:1::/48 nhg_id is 29
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_DELROUTE 2002:2222:2::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 2002:2222:2::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 2002:2222:2::/48 nhg_id is 29
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_DELROUTE 2003:3333:3::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 2003:3333:3::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 2003:3333:3::/48 nhg_id is 29
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_DELROUTE 2004:4444:4::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 2004:4444:4::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 2004:4444:4::/48 nhg_id is 29
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_DELROUTE 2005:5555:5::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 2005:5555:5::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 2005:5555:5::/48 nhg_id is 29
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_DELROUTE 2006:6666:6::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 2006:6666:6::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 2006:6666:6::/48 nhg_id is 29
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_DELROUTE 2007:7777:7::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 2007:7777:7::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 2007:7777:7::/48 nhg_id is 29
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_DELROUTE 2008:8888:8::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 2008:8888:8::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 2008:8888:8::/48 nhg_id is 29
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_DELROUTE 2009:9999:9::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 2009:9999:9::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 2009:9999:9::/48 nhg_id is 29
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_DELROUTE 200a:aaaa:a::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 200a:aaaa:a::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 200a:aaaa:a::/48 nhg_id is 29
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_DELROUTE 200b:bbbb:b::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 200b:bbbb:b::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 200b:bbbb:b::/48 nhg_id is 29
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_DELROUTE 200c:cccc:c::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 200c:cccc:c::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 200c:cccc:c::/48 nhg_id is 29
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_DELROUTE 200d:dddd:d::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 200d:dddd:d::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 200d:dddd:d::/48 nhg_id is 29
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_DELROUTE 200e:eeee:e::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 200e:eeee:e::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 200e:eeee:e::/48 nhg_id is 29
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_DELROUTE 200f:ffff:f::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 200f:ffff:f::/48 vrf 0(254)
2023/03/06 16:39:48 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 200f:ffff:f::/48 nhg_id is 29
2023/03/06 16:39:48 ZEBRA: [HYEHE-CQZ9G] nl_batch_send: netlink-dp (NS 0), batch size=1972, msg cnt=33
2023/03/06 16:39:54 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=88, seq=0, pid=0
2023/03/06 16:39:54 ZEBRA: [TDS34-MNEJW]     Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring
2023/03/06 16:39:54 ZEBRA: [KKAC1-JMWTB] Rx RTM_NEWNEIGH family ipv6 IF enp1s0(2) vrf default(0) IP fe80::6cd0:5cff:fee5:7820 MAC fe:54:00:20:30:bf state 0x4 flags 0x0 ext_flags 0x0
2023/03/06 16:40:29 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWLINK(16), len=1380, seq=0, pid=0
2023/03/06 16:40:29 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWLINK(16), len=1380, seq=0, pid=0
2023/03/06 16:40:29 ZEBRA: [W6BZR-YZPAB] RTM_NEWLINK update for enp7s0(4) sl_type 0 master 0 flags 0x11043
2023/03/06 16:40:29 ZEBRA: [N7FN2-J93A7] Intf enp7s0(4) has come UP
2023/03/06 16:40:29 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=116, seq=0, pid=0
2023/03/06 16:40:29 ZEBRA: [SKNFJ-G938V] RTM_NEWROUTE ipv6 multicast proto kernel NS 0
2023/03/06 16:40:29 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=116, seq=0, pid=0
2023/03/06 16:40:29 ZEBRA: [SKNFJ-G938V] RTM_NEWROUTE ipv6 unicast proto kernel NS 0
2023/03/06 16:40:31 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWADDR(20), len=72, seq=0, pid=0
2023/03/06 16:40:31 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=116, seq=0, pid=0
2023/03/06 16:40:31 ZEBRA: [SKNFJ-G938V] RTM_NEWROUTE ipv6 local proto kernel NS 0
2023/03/06 16:40:31 ZEBRA: [J3J81-V75NW] Route rtm_type: local(2) intentionally ignoring
2023/03/06 16:40:31 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-dp-in (NS 0) type RTM_NEWADDR(20), len=72, seq=0, pid=0
2023/03/06 16:40:31 ZEBRA: [RGWF1-EHXT1] netlink_interface_addr_dplane: RTM_NEWADDR nsid 0 ifindex 4 flags 0x80:
2023/03/06 16:40:31 ZEBRA: [ME3M2-X6YT9]   IFA_ADDRESS   fe80::5054:ff:fe1d:7b14/64
2023/03/06 16:40:31 ZEBRA: [P2VPT-508WP]   IFA_CACHEINFO pref -1, valid -1
2023/03/06 16:40:31 ZEBRA: [K8FXY-V65ZJ] Intf dplane ctx 0x7f58e8026e80, op INTF_ADDR_ADD, ifindex (4), result QUEUED
2023/03/06 16:40:31 ZEBRA: [MZPZA-W042K] zebra_if_addr_update_ctx: INTF_ADDR_ADD: ifindex enp7s0(4), addr fe80::5054:ff:fe1d:7b14/64
2023/03/06 16:40:31 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=88, seq=0, pid=0
2023/03/06 16:40:31 ZEBRA: [TDS34-MNEJW]     Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring
2023/03/06 16:40:31 ZEBRA: [KKAC1-JMWTB] Rx RTM_NEWNEIGH family ipv6 IF enp7s0(4) vrf default(0) IP fe80::5054:ff:fe13:76ca MAC 52:54:00:13:76:ca state 0x2 flags 0x0 ext_flags 0x0
2023/03/06 16:40:38 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=88, seq=0, pid=0
2023/03/06 16:40:38 ZEBRA: [TDS34-MNEJW]     Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring
2023/03/06 16:40:38 ZEBRA: [KKAC1-JMWTB] Rx RTM_NEWNEIGH family ipv6 IF enp7s0(4) vrf default(0) IP fe80::5054:ff:fe13:76ca MAC 52:54:00:13:76:ca state 0x2 flags 0x80 ext_flags 0x0
2023/03/06 16:40:38 ZEBRA: [HCFTH-1NSTK] netlink_talk: netlink-cmd (NS 0) type RTM_DELNEIGH(29), len=56 seq=15 flags 0x401
2023/03/06 16:40:38 ZEBRA: [MQ5AP-2S1F5] netlink-cmd (NS 0) error: No such file or directory, type=RTM_DELNEIGH(29), seq=15, pid=4288046248
2023/03/06 16:40:38 ZEBRA: [HCFTH-1NSTK] netlink_talk: netlink-cmd (NS 0) type RTM_NEWNEIGH(28), len=56 seq=16 flags 0x401
2023/03/06 16:40:38 BGP: [N9HHH-F8H1M] %ADJCHANGE: neighbor enp7s0(bgp-r2) in vrf default Up
2023/03/06 16:40:39 BGP: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for IPv6 Unicast from enp7s0 in vrf default
2023/03/06 16:40:39 BGP: [M59KS-A3ZXZ] bgp_update_receive: rcvd End-of-RIB for IPv4 Unicast from enp7s0 in vrf default
2023/03/06 16:40:39 ZEBRA: [YXCJP-0WZWV] netlink_nexthop_msg_encode: ID (34): fe80::5054:ff:fe13:76ca, via enp7s0(4) vrf default(0)  
2023/03/06 16:40:39 ZEBRA: [R43C6-KYHWT] netlink_nexthop_msg_encode: RTM_NEWNEXTHOP, id=34
2023/03/06 16:40:39 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 192.168.99.0/24 vrf 0(254)
2023/03/06 16:40:39 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 192.168.99.0/24 nhg_id is 33
2023/03/06 16:40:39 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_DELROUTE ::/0 vrf 0(254)
2023/03/06 16:40:39 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE ::/0 vrf 0(254)
2023/03/06 16:40:39 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: ::/0 nhg_id is 33
2023/03/06 16:40:39 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_DELROUTE 200f:ffff:f::/48 vrf 0(254)
2023/03/06 16:40:39 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 200f:ffff:f::/48 vrf 0(254)
2023/03/06 16:40:39 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 200f:ffff:f::/48 nhg_id is 33
2023/03/06 16:40:39 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_DELROUTE 200e:eeee:e::/48 vrf 0(254)
2023/03/06 16:40:39 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 200e:eeee:e::/48 vrf 0(254)
2023/03/06 16:40:39 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 200e:eeee:e::/48 nhg_id is 33
2023/03/06 16:40:39 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_DELROUTE 200d:dddd:d::/48 vrf 0(254)
2023/03/06 16:40:39 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 200d:dddd:d::/48 vrf 0(254)
2023/03/06 16:40:39 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 200d:dddd:d::/48 nhg_id is 33
2023/03/06 16:40:39 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_DELROUTE 200c:cccc:c::/48 vrf 0(254)
2023/03/06 16:40:39 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 200c:cccc:c::/48 vrf 0(254)
2023/03/06 16:40:39 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 200c:cccc:c::/48 nhg_id is 33
2023/03/06 16:40:39 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_DELROUTE 200b:bbbb:b::/48 vrf 0(254)
2023/03/06 16:40:39 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 200b:bbbb:b::/48 vrf 0(254)
2023/03/06 16:40:39 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 200b:bbbb:b::/48 nhg_id is 33
2023/03/06 16:40:39 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_DELROUTE 200a:aaaa:a::/48 vrf 0(254)
2023/03/06 16:40:39 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 200a:aaaa:a::/48 vrf 0(254)
2023/03/06 16:40:39 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 200a:aaaa:a::/48 nhg_id is 33
2023/03/06 16:40:39 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_DELROUTE 2009:9999:9::/48 vrf 0(254)
2023/03/06 16:40:39 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 2009:9999:9::/48 vrf 0(254)
2023/03/06 16:40:39 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 2009:9999:9::/48 nhg_id is 33
2023/03/06 16:40:39 ZEBRA: [HYEHE-CQZ9G] nl_batch_send: netlink-dp (NS 0), batch size=1072, msg cnt=18
2023/03/06 16:40:39 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_DELROUTE 2008:8888:8::/48 vrf 0(254)
2023/03/06 16:40:39 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 2008:8888:8::/48 vrf 0(254)
2023/03/06 16:40:39 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 2008:8888:8::/48 nhg_id is 33
2023/03/06 16:40:39 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_DELROUTE 2007:7777:7::/48 vrf 0(254)
2023/03/06 16:40:39 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 2007:7777:7::/48 vrf 0(254)
2023/03/06 16:40:39 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 2007:7777:7::/48 nhg_id is 33
2023/03/06 16:40:39 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_DELROUTE 2006:6666:6::/48 vrf 0(254)
2023/03/06 16:40:39 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 2006:6666:6::/48 vrf 0(254)
2023/03/06 16:40:39 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 2006:6666:6::/48 nhg_id is 33
2023/03/06 16:40:39 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_DELROUTE 2005:5555:5::/48 vrf 0(254)
2023/03/06 16:40:39 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 2005:5555:5::/48 vrf 0(254)
2023/03/06 16:40:39 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 2005:5555:5::/48 nhg_id is 33
2023/03/06 16:40:39 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_DELROUTE 2004:4444:4::/48 vrf 0(254)
2023/03/06 16:40:39 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 2004:4444:4::/48 vrf 0(254)
2023/03/06 16:40:39 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 2004:4444:4::/48 nhg_id is 33
2023/03/06 16:40:39 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_DELROUTE 2003:3333:3::/48 vrf 0(254)
2023/03/06 16:40:39 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 2003:3333:3::/48 vrf 0(254)
2023/03/06 16:40:39 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 2003:3333:3::/48 nhg_id is 33
2023/03/06 16:40:39 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_DELROUTE 2002:2222:2::/48 vrf 0(254)
2023/03/06 16:40:39 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 2002:2222:2::/48 vrf 0(254)
2023/03/06 16:40:39 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 2002:2222:2::/48 nhg_id is 33
2023/03/06 16:40:39 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_DELROUTE 2001:1111:1::/48 vrf 0(254)
2023/03/06 16:40:39 ZEBRA: [YXPF5-B2CE0] netlink_route_multipath_msg_encode: RTM_NEWROUTE 2001:1111:1::/48 vrf 0(254)
2023/03/06 16:40:39 ZEBRA: [J87BH-XW5PP] netlink_route_multipath_msg_encode: 2001:1111:1::/48 nhg_id is 33
2023/03/06 16:40:39 ZEBRA: [HYEHE-CQZ9G] nl_batch_send: netlink-dp (NS 0), batch size=960, msg cnt=16
2023/03/06 16:41:05 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=76, seq=0, pid=0
2023/03/06 16:41:05 ZEBRA: [TDS34-MNEJW]     Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring
2023/03/06 16:41:05 ZEBRA: [KKAC1-JMWTB] Rx RTM_NEWNEIGH family ipv4 IF enp2s0(3) vrf default(0) IP 192.168.122.1 MAC 52:54:00:21:58:d1 state 0x4 flags 0x0 ext_flags 0x0
2023/03/06 16:41:33 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=76, seq=0, pid=0
2023/03/06 16:41:33 ZEBRA: [TDS34-MNEJW]     Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring
2023/03/06 16:41:33 ZEBRA: [KKAC1-JMWTB] Rx RTM_NEWNEIGH family ipv4 IF enp2s0(3) vrf default(0) IP 192.168.122.1 MAC 52:54:00:21:58:d1 state 0x2 flags 0x0 ext_flags 0x0
2023/03/06 16:42:04 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=76, seq=0, pid=0
2023/03/06 16:42:04 ZEBRA: [TDS34-MNEJW]     Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring
2023/03/06 16:42:04 ZEBRA: [KKAC1-JMWTB] Rx RTM_NEWNEIGH family ipv4 IF enp2s0(3) vrf default(0) IP 192.168.122.1 MAC 52:54:00:21:58:d1 state 0x4 flags 0x0 ext_flags 0x0
2023/03/06 16:42:42 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=76, seq=0, pid=0
2023/03/06 16:42:42 ZEBRA: [TDS34-MNEJW]     Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring
2023/03/06 16:42:42 ZEBRA: [KKAC1-JMWTB] Rx RTM_NEWNEIGH family ipv4 IF enp2s0(3) vrf default(0) IP 192.168.122.1 MAC 52:54:00:21:58:d1 state 0x2 flags 0x0 ext_flags 0x0
2023/03/06 16:43:35 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=76, seq=0, pid=0
2023/03/06 16:43:35 ZEBRA: [TDS34-MNEJW]     Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring
2023/03/06 16:43:35 ZEBRA: [KKAC1-JMWTB] Rx RTM_NEWNEIGH family ipv4 IF enp2s0(3) vrf default(0) IP 192.168.122.1 MAC 52:54:00:21:58:d1 state 0x4 flags 0x0 ext_flags 0x0
2023/03/06 16:43:41 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=76, seq=0, pid=0
2023/03/06 16:43:41 ZEBRA: [TDS34-MNEJW]     Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring
2023/03/06 16:43:41 ZEBRA: [KKAC1-JMWTB] Rx RTM_NEWNEIGH family ipv4 IF enp2s0(3) vrf default(0) IP 192.168.122.1 MAC 52:54:00:21:58:d1 state 0x2 flags 0x0 ext_flags 0x0
2023/03/06 16:44:03 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=76, seq=0, pid=0
2023/03/06 16:44:03 ZEBRA: [TDS34-MNEJW]     Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring
2023/03/06 16:44:03 ZEBRA: [KKAC1-JMWTB] Rx RTM_NEWNEIGH family ipv4 IF enp2s0(3) vrf default(0) IP 192.168.122.1 MAC 52:54:00:21:58:d1 state 0x4 flags 0x0 ext_flags 0x0
2023/03/06 16:44:17 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=88, seq=0, pid=0
2023/03/06 16:44:17 ZEBRA: [TDS34-MNEJW]     Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring
2023/03/06 16:44:17 ZEBRA: [KKAC1-JMWTB] Rx RTM_NEWNEIGH family ipv6 IF enp7s0(4) vrf default(0) IP fe80::9cff:faff:fea1:2814 MAC fe:54:00:13:76:ca state 0x4 flags 0x0 ext_flags 0x0
2023/03/06 16:44:23 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=88, seq=0, pid=0
2023/03/06 16:44:23 ZEBRA: [TDS34-MNEJW]     Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring
2023/03/06 16:44:23 ZEBRA: [KKAC1-JMWTB] Rx RTM_NEWNEIGH family ipv6 IF enp7s0(4) vrf default(0) IP fe80::9cff:faff:fea1:2814 MAC fe:54:00:13:76:ca state 0x10 flags 0x0 ext_flags 0x0
2023/03/06 16:44:23 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=88, seq=0, pid=0
2023/03/06 16:44:23 ZEBRA: [TDS34-MNEJW]     Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring
2023/03/06 16:44:23 ZEBRA: [KKAC1-JMWTB] Rx RTM_NEWNEIGH family ipv6 IF enp1s0(2) vrf default(0) IP fe80::6cd0:5cff:fee5:7820 MAC fe:54:00:20:30:bf state 0x10 flags 0x0 ext_flags 0x0
2023/03/06 16:44:23 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=88, seq=0, pid=0
2023/03/06 16:44:23 ZEBRA: [TDS34-MNEJW]     Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring
2023/03/06 16:44:23 ZEBRA: [KKAC1-JMWTB] Rx RTM_NEWNEIGH family ipv6 IF enp7s0(4) vrf default(0) IP fe80::9cff:faff:fea1:2814 MAC fe:54:00:13:76:ca state 0x2 flags 0x0 ext_flags 0x0
2023/03/06 16:44:23 ZEBRA: [KMXEB-K771Y] netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=88, seq=0, pid=0
2023/03/06 16:44:23 ZEBRA: [TDS34-MNEJW]     Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring

I’m having a similar issue with 8.4.1. Setting zebra nexthop-group keep 1 seems to prevent it.

Hello all; I too believe I see this on several hosts with debian 12, kernel 6.1.38-1, frr 8.4.2-1 and 8.5.2-0. I have unnumbered bgp on top of vlans using netplan with provider systemd-networkd, in a clos architecture. I see behavior much like mwinter-osr describes when bouncing spine routers - leaf-to-leaf communication gets borked; 'show ip bgp' looks fine; kernel routes get out of sync with frr, ecmp nexthops never recover. The issue is tempermental overall.

edit: jetomit's zebra nexthop-group keep 1 fixes it for me. i can bounce my spines and all my ecmp routes recover.

There is a similar error going on at https://vyos.dev/T5424 (discussion at forum over at https://forum.vyos.io/t/frr-loses-routing-info-after-5-12k-l2tp-subs-connected/10422/ ) where zebra nexthop-group keep 1 didnt seem to be working as workaround for the situation.

Any suggestions for how to troubleshoot this further except for adding this to the config?

debug zebra kernel
debug zebra dplane
debug zebra dplane dpdk
debug zebra nexthop

I think recent commits might have fixed this ... can someone retest and see where we are off master?

Recent commits - you mean frr 9.1? I tried to build it under VyOS 1.4 on my Debian buster machine, but it requires libyang2-dev >=2.1.80 while buster has 2.1.30-2. I'm extremely interested to test it but I can do it on VyOS only.

Also current Debian stable 12.1 named bookworm (which 1.4-rolling is based on) use libyang2-dev (2.1.30-2) according to: https://packages.debian.org/bookworm/libyang2-dev

FRR 9.0 who is part of current 1.4-rolling (VyOS 1.4-rolling-202308060317 as of writing) is compiled with:

root@vyos:/home/vyos# vtysh

Hello, this is FRRouting (version 9.0).
Copyright 1996-2005 Kunihiro Ishiguro, et al.

vyos# show version
FRRouting 9.0 (vyos) on Linux(6.1.43-amd64-vyos).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
configured with:
    '--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-option-checking' '--disable-silent-rules' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--libexecdir=${prefix}/lib/x86_64-linux-gnu' '--disable-maintainer-mode' '--localstatedir=/var/run/frr' '--sbindir=/usr/lib/frr' '--sysconfdir=/etc/frr' '--with-vtysh-pager=/usr/bin/pager' '--libdir=/usr/lib/x86_64-linux-gnu/frr' '--with-moduledir=/usr/lib/x86_64-linux-gnu/frr/modules' '--disable-dependency-tracking' '--enable-rpki' '--disable-scripting' '--enable-pim6d' '--with-libpam' '--enable-doc' '--enable-doc-html' '--enable-snmp' '--enable-fpm' '--disable-protobuf' '--disable-zeromq' '--enable-ospfapi' '--enable-bgp-vnc' '--enable-multipath=256' '--enable-user=frr' '--enable-group=frr' '--enable-vty-group=frrvty' '--enable-configfile-mask=0640' '--enable-logfile-mask=0640' 'build_alias=x86_64-linux-gnu' 'PYTHON=python3'

Oh my fault. Definitely bookworm. Probably I need to know the branch name with the fix.. Seems that we can build the latest for VyOS The current is 9.1-dev-663-g8ceb26240

@mwinter-osr, @riw777, I can't reproduce the issue with the current master.

root@cl1:/frr# ip -6 ro show default
default nhid 139 proto bgp metric 20 pref medium
    nexthop via fe80::f8e8:fbff:fe1d:ce41 dev eth2 weight 1 
    nexthop via fe80::98f3:14ff:fe5a:1e7c dev eth1 weight 1

root@cl1:/frr# vtysh -c "show nexthop-group rib 139"
ID: 139 (zebra)
     RefCnt: 3
     Uptime: 00:04:15
     VRF: default
     Valid, Installed
     Depends: (132) (140)
           via fe80::98f3:14ff:fe5a:1e7c, eth1 (vrf default), weight 1
           via fe80::f8e8:fbff:fe1d:ce41, eth2 (vrf default), weight 1

root@cl1:/frr# ip next get id 139
id 139 group 132/140 proto zebra

root@cl1:/frr# ip link set down dev eth1
root@cl1:/frr# sleep 10
root@cl1:/frr# ip link set up dev eth1

root@cl1:/frr# vtysh -c "show ipv6 route ::/0 json" | grep nexthopGroupId
      "nexthopGroupId":147,

root@cl1:/frr# vtysh -c "show nexthop-group rib 147"
ID: 147 (zebra)
     RefCnt: 3
     Uptime: 00:00:35
     VRF: default
     Valid, Installed
     Depends: (132) (148)
           via fe80::98f3:14ff:fe5a:1e7c, eth1 (vrf default), weight 1
           via fe80::f8e8:fbff:fe1d:ce41, eth2 (vrf default), weight 1

root@cl1:/frr# ip -6 ro show default
default nhid 147 proto bgp metric 20 pref medium
    nexthop via fe80::f8e8:fbff:fe1d:ce41 dev eth2 weight 1 
    nexthop via fe80::98f3:14ff:fe5a:1e7c dev eth1 weight 1

root@cl1:/frr# ip next get id 147
id 147 group 132/148 proto zebra

Do we have any version where this is confirmed to be resolved based on the above?

FRRouting / frr