search

FRRouting / frr

The FRRouting Protocol Suite
https://frrouting.org/
Other
3.27k stars 1.24k forks source link

[9.2-dev] staticd/mgmtd doesn't load static routes on OS booting in a special case with more than 11 VRFs in system and wrong networking config #15267

Closed EasyNetDev closed 7 months ago

EasyNetDev commented 7 months ago

Describe the bug

When the system is booting up and you have 12 or more VRFs defined in system and in system networking you have a wrong configuration, FRRouting doesn't load static routes and are missing from the running configuration! I've notice this behavior from December on 4 PC and 1 VM running Debian 12 or 13 with latest development FRR. All of them have the same behavior.

To Reproduce

  1. Install latest git FRR.
  2. Create 11 or 12 VRFs in the system (example provided below).
  3. Create a wrong configuration interface in system configrau (example provided below).
  4. Under FRR add in each VRF and/or in GRT some routes (example provided below).
  5. Save the config.
  6. Reboot the machine.
  7. If you have 11 VRFs in system, routes are loaded in running config.
  8. If you have 12 or more VRFs in system, static routes are missing from running config.

Networking configuration:

  1. /etc/network/interfaces: 
    
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

source-directory /etc/network/interfaces.d/

The loopback network interface

auto lo iface lo inet loopback

The primary network interface

auto ens33 iface ens33 use dhcp

2. /etc/network/interfaces.d/01-vrf:

auto red iface red vrf-table 3000

auto red1 iface red1 vrf-table 3001

auto red2 iface red2 vrf-table 3002

auto red3 iface red3 vrf-table 3003

auto red4 iface red4 vrf-table 3004

auto red5 iface red5 vrf-table 3005

auto blue iface blue vrf-table 4000

auto blue1 iface blue1 vrf-table 4001

auto blue2 iface blue2 vrf-table 4002

auto blue3 iface blue3 vrf-table 4003

auto blue4 iface blue4 vrf-table 4004

auto blue5

iface blue5

vrf-table 4005

auto green iface green vrf-table 5000

auto green1

iface green1

vrf-table 5001

auto green2

iface green2

vrf-table 5002

auto green3

iface green3

vrf-table 5003

auto green4

iface green4

vrf-table 5004

auto green5

iface green5

vrf-table 5005


4. /etc/network/interfaces.d/05-loopback (with wrong configuration on purpose):

auto lo0 iface lo0 link-type dummy

auto lo1 iface lo1 link-type dummy vrf red10

auto lo2 iface lo2 link-type dummy vrf blue10


red10 and blue10 VRFs are not existing in system.

## Test with 12 VRFs:

grep ^auto /etc/network/interfaces.d/01-vrf | wc -l

12


### Output of networking.service and frr.service:

systemctl status networking

● networking.service - ifupdown-ng networking initialization Loaded: loaded (/usr/lib/systemd/system/networking.service; enabled; preset: enabled) Active: active (exited) since Wed 2024-01-31 23:05:03 CET; 9min ago Docs: man:interfaces(5) man:ifup(8) man:ifdown(8) Process: 673 ExecStart=/usr/share/ifupdown-ng/sbin/networking start (code=exited, status=0/SUCCESS) Main PID: 673 (code=exited, status=0/SUCCESS) Tasks: 1 (limit: 9435) Memory: 5.9M (peak: 9.2M) CPU: 799ms CGroup: /system.slice/networking.service └─1683 /usr/sbin/dhclient -pf /var/run/dhclient.ens33.pid ens33

Jan 31 23:05:03 FRR01 networking[691]: /bin/run-parts /etc/network/if-up.d Jan 31 23:05:03 FRR01 networking[691]: /usr/libexec/ifupdown-ng/link Jan 31 23:05:03 FRR01 networking[691]: /usr/libexec/ifupdown-ng/dhcp Jan 31 23:05:03 FRR01 systemd[1]: Finished networking.service - ifupdown-ng networking initialization. Jan 31 23:09:06 FRR01 dhclient[1683]: DHCPREQUEST for 192.168.55.187 on ens33 to 192.168.55.2 port 67 Jan 31 23:09:06 FRR01 dhclient[1683]: DHCPACK of 192.168.55.187 from 192.168.55.2 Jan 31 23:09:06 FRR01 dhclient[1683]: bound to 192.168.55.187 -- renewal in 287 seconds. Jan 31 23:13:53 FRR01 dhclient[1683]: DHCPREQUEST for 192.168.55.187 on ens33 to 192.168.55.2 port 67 Jan 31 23:13:53 FRR01 dhclient[1683]: DHCPACK of 192.168.55.187 from 192.168.55.2 Jan 31 23:13:53 FRR01 dhclient[1683]: bound to 192.168.55.187 -- renewal in 273 seconds.

systemctl status frr.service

● frr.service - FRRouting Loaded: loaded (/usr/lib/systemd/system/frr.service; enabled; preset: enabled) Active: active (running) since Wed 2024-01-31 23:05:03 CET; 10min ago Docs: https://frrouting.readthedocs.io/en/latest/setup.html Process: 657 ExecStart=/usr/lib/frr/frrinit.sh start (code=exited, status=0/SUCCESS) Main PID: 791 (watchfrr) Status: "FRR Operational" Tasks: 45 (limit: 9435) Memory: 124.3M (peak: 159.1M) CPU: 2.410s CGroup: /system.slice/frr.service ├─ 791 /usr/lib/frr/watchfrr -d -F traditional zebra mgmtd bgpd ospfd ospf6d isisd pimd pim6d ldpd nhrpd eigrpd pbrd stat> ├─ 879 /usr/lib/frr/zebra -d -F traditional -A 127.0.0.1 -s 90000000 -M snmp ├─ 931 /usr/lib/frr/mgmtd -d -F traditional -A 127.0.0.1 ├─1012 /usr/lib/frr/bgpd -d -F traditional -A 127.0.0.1 -M snmp ├─1060 /usr/lib/frr/ospfd -d -F traditional -A 127.0.0.1 -M snmp ├─1108 /usr/lib/frr/ospf6d -d -F traditional -A ::1 -M snmp ├─1159 /usr/lib/frr/isisd -d -F traditional -A 127.0.0.1 -M snmp ├─1204 /usr/lib/frr/pimd -d -F traditional -A 127.0.0.1 ├─1235 /usr/lib/frr/pim6d -d -F traditional -A ::1 ├─1254 /usr/lib/frr/ldpd -L -u frr -g frr ├─1255 /usr/lib/frr/ldpd -E -u frr -g frr ├─1275 /usr/lib/frr/ldpd -d -F traditional -A 127.0.0.1 -M snmp ├─1305 /usr/lib/frr/nhrpd -d -F traditional -A 127.0.0.1 ├─1331 /usr/lib/frr/eigrpd -d -F traditional -A 127.0.0.1 ├─1357 /usr/lib/frr/pbrd -d -F traditional -A 127.0.0.1 ├─1383 /usr/lib/frr/staticd -d -F traditional -A 127.0.0.1 ├─1452 /usr/lib/frr/bfdd -d -F traditional -A 127.0.0.1 ├─1474 /usr/lib/frr/vrrpd -d -F traditional -A 127.0.0.1 └─1502 /usr/lib/frr/pathd -d -F traditional -A 127.0.0.1

Jan 31 23:05:03 FRR01 watchfrr[791]: [QDG3Y-BY5TN] staticd state -> up : connect succeeded Jan 31 23:05:03 FRR01 watchfrr[791]: [QDG3Y-BY5TN] bfdd state -> up : connect succeeded Jan 31 23:05:03 FRR01 watchfrr[791]: [QDG3Y-BY5TN] vrrpd state -> up : connect succeeded Jan 31 23:05:03 FRR01 watchfrr[791]: [QDG3Y-BY5TN] pathd state -> up : connect succeeded Jan 31 23:05:03 FRR01 watchfrr[791]: [KWE5Q-QNGFC] all daemons up, doing startup-complete notify Jan 31 23:05:03 FRR01 frrinit.sh[657]: Started watchfrr. Jan 31 23:05:03 FRR01 systemd[1]: Started frr.service - FRRouting. Jan 31 23:05:03 FRR01 zebra[879]: [V98V0-MTWPF] client 130 says hello and bids fair to announce only bgp routes vrf=0 Jan 31 23:05:06 FRR01 pathd[1502]: [NN4XW-E4M3V] IPv6 Router Id updated for VRF 0: fc00:192:168:55:20c:29ff:fee4:52f Jan 31 23:05:06 FRR01 zebra[879]: [RHHTS-F96DR][EC 4043309149] ens33(2): Rx RA - our AdvManagedFlag (0) doesn't agree with fe80::58c1:>


### FRR running config:

vtysh -c "show run"

Building configuration...

Current configuration: ! frr version 9.2-dev frr defaults traditional hostname FRR01 log syslog informational no ip forwarding no ipv6 forwarding service integrated-vtysh-config ! vrf blue exit-vrf ! vrf blue1 exit-vrf ! vrf blue2 exit-vrf ! vrf blue3 exit-vrf ! vrf blue4 exit-vrf ! vrf green exit-vrf ! vrf red exit-vrf ! vrf red1 exit-vrf ! vrf red2 exit-vrf ! vrf red3 exit-vrf ! vrf red4 exit-vrf ! vrf red5 exit-vrf ! router bgp 65000 bgp router-id 10.100.1.1 no bgp suppress-duplicates bgp graceful-shutdown bgp graceful-restart neighbor MPLS-v4 peer-group neighbor MPLS-v4 remote-as 65000 neighbor MPLS-v4 description Core Routers MPLS ...


## Test with 11 VRFs:

grep ^auto /etc/network/interfaces.d/01-vrf | wc -l

11


### Output of networking.service and frr.service:

systemctl status networking.service

● networking.service - ifupdown-ng networking initialization Loaded: loaded (/usr/lib/systemd/system/networking.service; enabled; preset: enabled) Active: active (exited) since Wed 2024-01-31 23:34:23 CET; 1min 28s ago Docs: man:interfaces(5) man:ifup(8) man:ifdown(8) Process: 730 ExecStart=/usr/share/ifupdown-ng/sbin/networking start (code=exited, status=0/SUCCESS) Main PID: 730 (code=exited, status=0/SUCCESS) Tasks: 1 (limit: 9435) Memory: 5.7M (peak: 8.5M) CPU: 749ms CGroup: /system.slice/networking.service └─1678 /usr/sbin/dhclient -pf /var/run/dhclient.ens33.pid ens33

Jan 31 23:34:23 FRR01 networking[753]: /usr/libexec/ifupdown-ng/dhcp Jan 31 23:34:23 FRR01 networking[753]: /usr/libexec/ifupdown-ng/link Jan 31 23:34:23 FRR01 networking[753]: /usr/libexec/ifupdown-ng/dhcp Jan 31 23:34:23 FRR01 networking[753]: /bin/run-parts /etc/network/if-pre-up.d Jan 31 23:34:23 FRR01 networking[753]: /usr/libexec/ifupdown-ng/link Jan 31 23:34:23 FRR01 networking[753]: /usr/libexec/ifupdown-ng/dhcp Jan 31 23:34:23 FRR01 networking[753]: /bin/run-parts /etc/network/if-up.d Jan 31 23:34:23 FRR01 networking[753]: /usr/libexec/ifupdown-ng/link Jan 31 23:34:23 FRR01 networking[753]: /usr/libexec/ifupdown-ng/dhcp Jan 31 23:34:23 FRR01 systemd[1]: Finished networking.service - ifupdown-ng networking initialization.

systemctl status frr.service

● frr.service - FRRouting Loaded: loaded (/usr/lib/systemd/system/frr.service; enabled; preset: enabled) Active: active (running) since Wed 2024-01-31 23:34:23 CET; 2min 22s ago Docs: https://frrouting.readthedocs.io/en/latest/setup.html Process: 717 ExecStart=/usr/lib/frr/frrinit.sh start (code=exited, status=0/SUCCESS) Main PID: 833 (watchfrr) Status: "FRR Operational" Tasks: 45 (limit: 9435) Memory: 124.7M (peak: 169.0M) CPU: 1.615s CGroup: /system.slice/frr.service ├─ 833 /usr/lib/frr/watchfrr -d -F traditional zebra mgmtd bgpd ospfd ospf6d isisd pimd pim6d ldpd nhrpd eigrpd pbrd stat> ├─ 929 /usr/lib/frr/zebra -d -F traditional -A 127.0.0.1 -s 90000000 -M snmp ├─ 979 /usr/lib/frr/mgmtd -d -F traditional -A 127.0.0.1 ├─1060 /usr/lib/frr/bgpd -d -F traditional -A 127.0.0.1 -M snmp ├─1109 /usr/lib/frr/ospfd -d -F traditional -A 127.0.0.1 -M snmp ├─1153 /usr/lib/frr/ospf6d -d -F traditional -A ::1 -M snmp ├─1205 /usr/lib/frr/isisd -d -F traditional -A 127.0.0.1 -M snmp ├─1251 /usr/lib/frr/pimd -d -F traditional -A 127.0.0.1 ├─1289 /usr/lib/frr/pim6d -d -F traditional -A ::1 ├─1301 /usr/lib/frr/ldpd -L -u frr -g frr ├─1302 /usr/lib/frr/ldpd -E -u frr -g frr ├─1328 /usr/lib/frr/ldpd -d -F traditional -A 127.0.0.1 -M snmp ├─1353 /usr/lib/frr/nhrpd -d -F traditional -A 127.0.0.1 ├─1378 /usr/lib/frr/eigrpd -d -F traditional -A 127.0.0.1 ├─1405 /usr/lib/frr/pbrd -d -F traditional -A 127.0.0.1 ├─1423 /usr/lib/frr/staticd -d -F traditional -A 127.0.0.1 ├─1477 /usr/lib/frr/bfdd -d -F traditional -A 127.0.0.1 ├─1504 /usr/lib/frr/vrrpd -d -F traditional -A 127.0.0.1 └─1535 /usr/lib/frr/pathd -d -F traditional -A 127.0.0.1

Jan 31 23:34:24 FRR01 zebra[929]: [V98V0-MTWPF] client 130 says hello and bids fair to announce only bgp routes vrf=0 Jan 31 23:34:25 FRR01 staticd[1423]: [NZTXV-H7AQ5] route_notify_owner: Route ::/0 over-ridden by better route for table: 254 Jan 31 23:34:26 FRR01 pathd[1535]: [NN4XW-E4M3V] IPv6 Router Id updated for VRF 0: fc00:192:168:55:20c:29ff:fee4:52f Jan 31 23:34:27 FRR01 zebra[929]: [RHHTS-F96DR][EC 4043309149] ens33(2): Rx RA - our AdvManagedFlag (0) doesn't agree with fe80::58c1:> Jan 31 23:34:28 FRR01 zebra[929]: [WVJCK-PPMGD][EC 4043309093] netlink-dp (NS 0) error: No such device, type=RTM_NEWROUTE(24), seq=48,> Jan 31 23:34:28 FRR01 staticd[1423]: [NZTXV-H7AQ5] route_notify_owner: Route 0.0.0.0/0 over-ridden by better route for table: 5000 Jan 31 23:34:28 FRR01 zebra[929]: [WVJCK-PPMGD][EC 4043309093] netlink-dp (NS 0) error: No such device, type=RTM_NEWROUTE(24), seq=51,> Jan 31 23:34:28 FRR01 staticd[1423]: [NZTXV-H7AQ5] route_notify_owner: Route ::/0 over-ridden by better route for table: 5000 Jan 31 23:34:28 FRR01 zebra[929]: [VYKYC-709DP] green(26:5000):0.0.0.0/0: Route install failed Jan 31 23:34:28 FRR01 zebra[929]: [VYKYC-709DP] green(26:5000):::/0: Route install failed


### FRR running config:

frr version 9.2-dev frr defaults traditional hostname FRR01 log syslog informational no ip forwarding no ipv6 forwarding service integrated-vtysh-config ! ip route 0.0.0.0/0 blackhole 250 ip route 192.168.88.0/24 192.168.55.1 ipv6 route ::/0 blackhole 250 ! vrf red ip route 0.0.0.0/0 blackhole 250 ipv6 route ::/0 blackhole 250 exit-vrf ! vrf blue ip route 0.0.0.0/0 blackhole 250 ipv6 route ::/0 blackhole 250 exit-vrf ! vrf green ip route 0.0.0.0/0 blackhole 250 ipv6 route ::/0 blackhole 250 exit-vrf ! vrf blue1 exit-vrf ! vrf blue2 exit-vrf ! vrf blue3 exit-vrf ! vrf red1 exit-vrf ! vrf red2 exit-vrf ! vrf red3 exit-vrf ! vrf red4 exit-vrf ! vrf red5 exit-vrf ! router bgp 65000 bgp router-id 10.100.1.1 no bgp suppress-duplicates bgp graceful-shutdown bgp graceful-restart neighbor MPLS-v4 peer-group ...


## Saved config that should be loaded at boot:

cat /etc/frr/frr.conf

frr version 9.2-dev frr defaults traditional hostname FRR01 log syslog informational no ip forwarding no ipv6 forwarding service integrated-vtysh-config ! ip route 0.0.0.0/0 blackhole 250 ip route 192.168.88.0/24 192.168.55.1 ipv6 route ::/0 blackhole 250 ! vrf red ip route 0.0.0.0/0 blackhole 250 ipv6 route ::/0 blackhole 250 exit-vrf ! vrf blue ip route 0.0.0.0/0 blackhole 250 ipv6 route ::/0 blackhole 250 exit-vrf ! vrf green ip route 0.0.0.0/0 blackhole 250 ipv6 route ::/0 blackhole 250 exit-vrf ! router bgp 65000 bgp router-id 10.100.1.1 no bgp suppress-duplicates bgp graceful-shutdown bgp graceful-restart neighbor MPLS-v4 peer-group neighbor MPLS-v4 remote-as 65000 neighbor MPLS-v4 description Core Routers MPLS neighbor MPLS-v4 bfd neighbor MPLS-v4 bfd check-control-plane-failure no neighbor MPLS-v4 enforce-first-as neighbor MPLS-v4 update-source lo neighbor MPLS-v4 capability extended-nexthop neighbor MPLS-v6 peer-group neighbor MPLS-v6 remote-as 65000 neighbor MPLS-v6 description Core Routers MPLS neighbor MPLS-v6 bfd neighbor MPLS-v6 bfd check-control-plane-failure no neighbor MPLS-v6 enforce-first-as neighbor MPLS-v6 update-source lo neighbor MPLS-v6 capability extended-nexthop neighbor SPOKES-v4 peer-group neighbor SPOKES-v4 remote-as 65000 neighbor SPOKES-v4 description SPOKES neighbor SPOKES-v4 bfd neighbor SPOKES-v4 bfd check-control-plane-failure neighbor SPOKES-v4 graceful-shutdown no neighbor SPOKES-v4 enforce-first-as neighbor SPOKES-v4 update-source lo neighbor SPOKES-v4 capability extended-nexthop neighbor SPOKES-v4 graceful-restart neighbor SPOKES-v6 peer-group neighbor SPOKES-v6 remote-as 65000 neighbor SPOKES-v6 description SPOKES neighbor SPOKES-v6 bfd neighbor SPOKES-v6 bfd check-control-plane-failure neighbor SPOKES-v6 graceful-shutdown no neighbor SPOKES-v6 enforce-first-as neighbor SPOKES-v6 update-source lo neighbor SPOKES-v6 capability extended-nexthop neighbor SPOKES-v6 graceful-restart neighbor 10.100.2.1 peer-group MPLS-v4 neighbor 10.100.2.1 description R02_VPNv4 no neighbor 10.100.2.1 enforce-first-as neighbor fc00:0:0:2::1 peer-group MPLS-v6 neighbor fc00:0:0:2::1 description R02_VPNv6 neighbor fc00:0:0:2::1 bfd no neighbor fc00:0:0:2::1 enforce-first-as neighbor fc00:0:0:2::1 graceful-restart ! address-family ipv4 unicast redistribute connected route-map VPN-export-GRT-connected no neighbor MPLS-v4 activate no neighbor MPLS-v6 activate no neighbor SPOKES-v4 activate no neighbor SPOKES-v6 activate label vpn export auto rd vpn export 43474:11001 rt vpn import 43474:11000 43474:11999 rt vpn export 43474:11000 export vpn import vpn exit-address-family ! address-family ipv4 multicast network 239.0.0.0/8 exit-address-family ! address-family ipv4 vpn neighbor MPLS-v4 activate neighbor SPOKES-v4 activate neighbor SPOKES-v4 soft-reconfiguration inbound neighbor SPOKES-v4 route-map to-SPOKES out exit-address-family ! address-family ipv6 unicast redistribute connected route-map VPN-export-GRT-connected label vpn export auto rd vpn export 43474:11001 rt vpn import 43474:11000 43474:11999 rt vpn export 43474:11000 export vpn import vpn exit-address-family ! address-family ipv6 vpn neighbor MPLS-v6 activate neighbor SPOKES-v6 activate neighbor SPOKES-v6 soft-reconfiguration inbound neighbor SPOKES-v6 route-map to-SPOKES out exit-address-family exit ! router bgp 65000 vrf red bgp router-id 10.100.1.12 ! address-family ipv4 unicast redistribute connected route-map VPN-export-RED-connected redistribute static route-map VPN-export-RED-static label vpn export auto rd vpn export 65000:16001 rt vpn import 65000:16000 65000:16100 65000:16200 65000:16800 65000:16999 rt vpn export 65000:16000 export vpn import vpn exit-address-family ! address-family ipv6 unicast redistribute connected route-map VPN-export-RED-connected redistribute static route-map VPN-export-RED-static label vpn export auto rd vpn export 65000:16001 rt vpn import 65000:16000 65000:16100 65000:16200 65000:16800 65000:16999 rt vpn export 65000:16000 export vpn import vpn exit-address-family exit ! router bgp 65000 vrf blue bgp router-id 10.100.1.16 ! address-family ipv4 unicast redistribute connected route-map VPN-export-BLUE-connected redistribute static route-map VPN-export-BLUE-static label vpn export auto rd vpn export 65000:12001 rt vpn import 65000:12000 65000:12100 65000:12200 65000:12800 65000:12999 rt vpn export 65000:12000 export vpn import vpn exit-address-family ! address-family ipv6 unicast redistribute connected route-map VPN-export-BLUE-connected redistribute static route-map VPN-export-BLUE-static label vpn export auto rd vpn export 65000:12001 rt vpn import 65000:12000 65000:12100 65000:12200 65000:12800 65000:12999 rt vpn export 65000:12000 export vpn import vpn exit-address-family exit ! router bgp 65000 vrf green bgp router-id 10.100.1.11 ! address-family ipv4 unicast redistribute connected route-map VPN-export-GREEN-connected label vpn export auto rd vpn export 65000:14001 rt vpn import 65000:14000 65000:14100 65000:14200 65000:14800 65000:14999 rt vpn export 65000:14000 export vpn import vpn exit-address-family ! address-family ipv6 unicast redistribute connected route-map VPN-export-GREEN-connected label vpn export auto rd vpn export 65000:14001 rt vpn import 65000:14000 65000:14100 65000:14200 65000:14800 65000:14999 rt vpn export 65000:14000 export vpn import vpn exit-address-family ! address-family l2vpn evpn advertise ipv4 unicast exit-address-family exit ! router ospf ospf router-id 10.100.1.1 log-adjacency-changes compatible rfc1583 maximum-paths 4 flood-reduction fast-reroute ti-lfa area 10 stub no-summary mpls ldp-sync exit ! router ospf vrf iptv ospf router-id 10.100.1.1 log-adjacency-changes compatible rfc1583 maximum-paths 4 neighbor 10.190.129.4 default-information originate always exit ! router ospf6 ospf6 router-id 10.100.1.1 exit ! mpls ldp router-id 10.100.1.1 dual-stack cisco-interop ordered-control neighbor 10.100.2.1 ttl-security disable neighbor 10.100.2.1 password ENPDW ! address-family ipv4 discovery transport-address 10.100.1.1 ! interface lan0.3001 exit ! interface wan0.3000 exit ! exit-address-family ! address-family ipv6 discovery transport-address fc00:0:0:1::1 ! interface lan0.3001 exit ! interface wan0.3000 exit ! exit-address-family ! exit ! router isis EasyNet net 49.0001.0101.9000.0001.00 fast-reroute load-sharing disable level-1 fast-reroute lfa tiebreaker lowest-backup-metric index 1 level-1 fast-reroute load-sharing disable level-2 exit ! router isis EasyNet-A2 net 49.0002.0101.9000.0001.00 lsp-mtu 1400 exit ! route-map to-SPOKES deny 60000 exit ! route-map VPN-export-GRT-connected permit 1000 match ip address prefix-list pl-EASYNET-subnets set extcommunity rt 65000:10999 65000:18999 65000:17999 65000:13999 65000:12999 65000:11000 65000:14999 65000:15999 exit ! route-map VPN-export-GRT-connected permit 1100 match ipv6 address prefix-list pl-EASYNET-subnets set extcommunity rt 65000:10999 65000:18999 65000:17999 65000:13999 65000:12999 65000:11000 65000:14999 65000:15999 exit ! route-map VPN-export-GRT-connected permit 2000 match ip address prefix-list pl-INTERCONNECT-subnets set extcommunity rt 65000:12999 65000:15999 exit ! route-map VPN-export-GRT-connected permit 2100 match ipv6 address prefix-list pl-INTERCONNECT-subnets set extcommunity rt 65000:12999 65000:15999 exit ! route-map VPN-export-GRT-connected deny 10000 exit ! route-map VPN-export-RED-connected permit 1000 set extcommunity rt 65000:16100 65000:16800 65000:12999 65000:15999 65000:18999 65000:17999 exit ! route-map VPN-export-RED-static permit 1000 set extcommunity rt 65000:16100 65000:16800 65000:12999 65000:15999 65000:18999 65000:17999 exit ! route-map VPN-export-BLUE-connected permit 1000 match ip address prefix-list pl-EASYNET-subnets match source-protocol local set extcommunity rt 65000:12100 65000:10999 65000:14999 65000:15999 65000:16999 65000:17999 exit ! route-map VPN-export-BLUE-connected permit 1100 match ipv6 address prefix-list pl-EASYNET-subnets set extcommunity rt 65000:12100 65000:10999 65000:14999 65000:15999 65000:16999 65000:17999 exit ! route-map VPN-export-BLUE-static deny 1000 match ip address prefix-list pl-DEFAULT-ROUTE exit ! route-map VPN-export-BLUE-static deny 1100 match ipv6 address prefix-list pl-DEFAULT-ROUTE exit ! route-map VPN-export-BLUE-static permit 2000 set extcommunity rt 65000:12200 65000:12800 65000:10999 65000:11999 65000:14999 65000:15999 65000:16999 exit ! route-map VPN-export-GREEN-connected permit 1000 set extcommunity rt 65000:14100 65000:14800 65000:12999 65000:15999 65000:18999 65000:17999 exit ! segment-routing traffic-eng exit exit !



**Expected behavior**
<!--
Write here a clear and concise description of what you expected to happen when
using the reproduction steps you provided above
-->
All static routes should be installed on boot.
To fix the issue I have to run `vtysh -d mgmtd -f /etc/frr/frr.conf` after booting.

**Screenshots**
<!--
If applicable, add screenshots to help explain your problem.
-->

**Versions**
<!--
Include your operating system type and version here

FAILURE TO PROVIDE THIS MAY RESULT IN YOUR ISSUE BEING IGNORED
-->
<!-- e.g. Fedora 24, Debian 10] -->
 - OS Version: Debian 13 Trixie
<!-- [e.g. Linux 5.4, OpenBSD 6.6] -->
 - Kernel: 6.5.0-5-amd64
<!-- e.g. 6.0, 7.4 -->
 - FRR Version: 9.2-dev
 - ifupdown-ng 0.13.0 using patched with PRs for VRRP, Teaming and CRC32 tool for VRRP MACVLAN permanent naming: https://github.com/EasyNetDev/ifupdown-ng/tree/main and Debian dpkg-buildpackage: https://github.com/EasyNetDev/ifupdown-ng/tree/ifupdown-ng/debian

**Additional context**
<!--
Add any other context about the problem here.
-->
EasyNetDev commented 7 months ago

To solve the booting issue I've created a service like this:

/etc/systemd/system/frr-static-routes-fix.service:

[Unit]
Description=FRRouting static routes fix
Documentation=https://frrouting.readthedocs.io/en/latest/setup.html
Documentation=https://github.com/FRRouting/frr/issues/15267
Wants=network.target
Requires=frr.service
After=network-pre.target systemd-sysctl.service frr.service
Before=network.target

[Service]
Nice=-5
Type=oneshot
RemainAfterExit=yes
#NotifyAccess=all
StartLimitInterval=3m
StartLimitBurst=3
TimeoutSec=2m
WatchdogSec=60s
RestartSec=5
#Restart=always
ExecStart=/usr/bin/vtysh -d mgmtd -f "/etc/frr/frr.conf"

[Install]
WantedBy=multi-user.target
idryzhov commented 7 months ago

Hi. Would be great if you could get and test the branch from this PR: https://github.com/FRRouting/frr/pull/15286. I think it should fix your problem.

EasyNetDev commented 7 months ago

Hi @idryzhov I will try to patch FRR with that PR and I will give a try. Thanks!

EasyNetDev commented 7 months ago

@idryzhov I compiled today's git FRR with your patch. I had to change a little bit the patch to be able to applied on the latest dev.

Looks like is working now.

idryzhov commented 7 months ago

Perfect. Thanks for the testing!

EasyNetDev commented 7 months ago

I will test on all devices tomorrow