Unable to set default route under certain conditions

socs commented 1 month ago

Current Behavior

Add a default route with some metric via shell, eg using ip route. Now running cli and listing routes will trigger internal error and trying to configure a default route will silently fail.

admin@infix:~$ sudo ip route add default dev br0 metric 10
admin@infix:~$ ip r s
default dev br0 scope link metric 10
10.10.60.0/24 dev e28 proto kernel scope link src 10.10.60.218
169.254.0.0/30 dev eth1 proto kernel scope link src 169.254.0.1
169.254.0.0/16 dev br0 proto kernel scope link src 169.254.1.1
admin@infix:~$ cli

See the 'help' command for an introduction to the system

admin@infix:/> show routes
[ERR] User callback failed.
sysrepocfg error: Internal error (Internal error)
For more details you may try to increase the verbosity up to "-v3".
sysrepocfg error: User callback failed. (User callback failed)
For more details you may try to increase the verbosity up to "-v3".
sysrepocfg error: Getting data failed (User callback failed)
For more details you may try to increase the verbosity up to "-v3".
Error, invalid JSON input
admin@infix:/> configure
admin@infix:/config/> set routing control-plane-protocol static name default ipv4 route 0.0.0.0/0 next-hop next-hop-address 10.10.60.1
admin@infix:/config/> leave
admin@infix:/> show routes
[ERR] User callback failed.
...
admin@infix:/> exit
admin@infix:~$ ip r s
default dev br0 scope link metric 10
10.10.60.0/24 dev e28 proto kernel scope link src 10.10.60.218
169.254.0.0/30 dev eth1 proto kernel scope link src 169.254.0.1
169.254.0.0/16 dev br0 proto kernel scope link src 169.254.1.1
admin@infix:~$ cli

See the 'help' command for an introduction to the system

admin@infix:/> configure
admin@infix:/config/> show routing control-plane-protocol static
static-routes {
  ipv4 {
    route 0.0.0.0/0 {
      next-hop {
        next-hop-address 10.10.60.1;
      }
    }
  }
}

Expected Behavior

Should be able to set default route. Should be able to list routes.

Steps To Reproduce

infix @ f77f585 installed.

admin@infix:~$ sudo ip route add default dev br0 metric 10
admin@infix:~$ klish -c "show routes"

Additional information

No response

troglobit commented 1 month ago

Reproduced. We can also see what the root cause is. Working on a fix ...

troglobit commented 1 month ago

Root cause is the metric which avahi-autoipd and udhcpc write their routes with to the kernel. The value is too low compared to the values calculated by Frr. Frr calculate the kernel metric using protocol distance + protol metric, where the distance is the high order byte in an unsigned 32-bit kernel metric.

3-part fix required, first part in progress (fix avahi-autoipd.action script and udhcpc.script). Second and third parts require a bit more work and will be handled separately, probably targeting the 24.10 milestone.

troglobit commented 1 month ago

Sorry for the delay.

Pull request #639 addresses this issue by a complete overhaul of how routes are managed in Infix. This was initially planned for v24.10, but the constraints on the "quick fix" would've been too difficult to explain, so the correct fix™ was pulled forward.

All routes, including configured/static, acquired from DHCP client (option 121 or 3), or IPv4LL (ZeroConf) are now installed into Frr via the standard staticd daemon.
Integration in staticd is by means of an /etc/frr/static.d/*.conf which can be inspected freely at runtime from the shell
While the kernel routing table use metrics, Frr use distance between routing protocols
Infix now use distance in the IETF standard guise of routing preference, lower numeric value has higher priority
Kernel routes (only connected routes now) are set with distance 0
Routes from DHCP client are set with distance 5 by default, override possible per client-if using the route-preference setting
Routes from IPv4LL (ZeroConf), i.e., connected default route, are always set with distance 254
Static routes are set using the default Cisco/Frr distance 1, override possible per route using the route-preference setting
OSPF routes are set with distance 110, i.e., can be overridden using static routes
All active RIB and FIB routes are now read from Frr instead of the kernel, see doc/networking.md for details

Finally, the CLI show routes command has been greatly improved with a show ip route Cisco style look, see the documentation for details, here's an example:

admin@R1:~$ show ip route
   PREFIX                PREF  NEXT-HOP     PROTOCOL    UPTIME
   10.0.0.1/32       110/2000  0.0.0.0      ospfv2     0h0m37s
>* 10.0.0.1/32            0/0  e5           direct     0h0m37s
>* 10.0.0.2/32          110/1  10.0.12.2    ospfv2     0h0m22s
>* 10.0.0.3/32       110/2000  10.0.13.2    ospfv2     0h0m21s
>* 10.0.0.4/32       110/2001  10.0.12.2    ospfv2     0h0m16s
   10.0.12.0/30         110/1  e6           ospfv2     0h0m37s
>* 10.0.12.0/30           0/0  e6           direct     0h0m37s
   10.0.13.0/30      110/2000  e5           ospfv2     0h0m37s
>* 10.0.13.0/30           0/0  e5           direct     0h0m37s
>* 10.0.23.0/30      110/2001  10.0.13.2    ospfv2     0h0m21s
>* 10.0.24.0/30      110/2001  10.0.12.2    ospfv2     0h0m16s
>* 10.0.41.0/30      110/2002  10.0.12.2    ospfv2     0h0m16s
>* 11.0.8.0/24            0/0  lo           direct     0h0m37s
>* 11.0.8.1/32          110/0  lo           ospfv2     0h0m37s
>* 11.0.9.1/32          110/1  10.0.12.2    ospfv2     0h0m22s
>* 11.0.10.1/32         110/1  10.0.12.2    ospfv2     0h0m22s
>* 11.0.11.1/32         110/1  10.0.12.2    ospfv2     0h0m22s
>* 11.0.12.1/32         110/1  10.0.12.2    ospfv2     0h0m22s
>* 11.0.13.1/32         110/1  10.0.12.2    ospfv2     0h0m22s
>* 11.0.14.1/32         110/1  10.0.12.2    ospfv2     0h0m22s
>* 11.0.15.1/32         110/1  10.0.12.2    ospfv2     0h0m22s
>* 192.168.3.0/24    110/2001  10.0.13.2    ospfv2     0h0m21s
>* 192.168.4.0/24      110/20  10.0.12.2    ospfv2     0h0m15s

socs commented 1 month ago

Sorry for the delay.

You're quite modest, pretty sure you broke some record on this one. Outstanding work

kernelkit / infix