Prefix delegation deletes kernel-created interface routes on FreeBSD

[ppaeps]

One of my networks has a static ULA prefix configuration in addition to DHCPv6 prefix delegation. When dhcpcd receives a prefix, it deletes the ULA interface routes.

Before running dhcpcd:

% netstat -rn -finet6 |grep fd4a:9620:34ad
fd4a:9620:34ad:1::/64             link#4                        U      vlan1
fd4a:9620:34ad:1:xxxx:xxxx:xxxx:xxxx link#4                     UHS         lo0
fd4a:9620:34ad:2::/64             link#5                        U      vlan2
fd4a:9620:34ad:2:xxxx:xxxx:xxxx:xxxx link#5                     UHS         lo0
fd4a:9620:34ad:3::/64             link#6                        U      vlan3
fd4a:9620:34ad:3:xxxx:xxxx:xxxx:xxxx link#6                     UHS         lo0

After running dhcpcd:

% netstat -rn -finet6 |grep fd4a:9620:34ad
fd4a:9620:34ad:1:xxxx:xxxx:xxxx:xxxx link#4                     UHS         lo0
fd4a:9620:34ad:2:xxxx:xxxx:xxxx:xxxx link#5                     UHS         lo0
fd4a:9620:34ad:3:xxxx:xxxx:xxxx:xxxx link#6                     UHS         lo0

Consequently, the system can no longer communicate with its neighbours using ULA addresses.

This system is running FreeBSD 12.2-RELEASE-p7.

rc.conf:

ifconfig_bge0="up"
vlans_bge0="vlan1 vlan2 vlan3"
create_args_vlan1="vlan 1"
create_args_vlan2="vlan 2"
create_args_vlan3="vlan 3"

ifconfig_bge1="up"
ifconfig_bge1_ipv6="inet6 auto_linklocal"
ipv6_cpe_wanif="bge1"

ifconfig_vlan1_ipv6="inet6 auto_linklocal"
ifconfig_vlan1_alias0="inet6 fd4a:9620:34ad:1:: eui64 prefixlen 64"
ifconfig_vlan2_ipv6="inet6 auto_linklocal"
ifconfig_vlan2_alias0="inet6 fd4a:9620:34ad:2:: eui64 prefixlen 64"
ifconfig_vlan3_ipv6="inet6 auto_linklocal"
ifconfig_vlan3_alias0="inet6 fd4a:9620:34ad:3:: eui64 prefixlen 64"

dhcpcd_enable="YES"

dhcpcd.conf:

noipv6rs
allowinterfaces bge1
interface bge1
  ipv6rs
  ia_na 1
  ia_pd 2 vlan1/1/64/0 vlan2/2/64/0 vlan3/3/64/0

[rsmarples]

Can you add this to dhcpcd.conf please to get some context?

debug
logfile /var/log/dhcpcd.log

And then paste the log here. I'm specifically interested in seeing what prefix dhcpcd gets AND the exact text dhcpcd logs when deleting your static routes. The only time dhcpcd should delete an existing route is if it collides with a route it needs to add.

[ppaeps]

That does not seem to produce any additional logging. No /var/log/dhcpcd is created. I suspect this might be something to do with sandboxing. I tried running dhcpcd -d and keeping an eye on stderr and /var/log/debug.log but nothing meaningful seems to be logged.

dhcpcd-9.4.0 starting
chrooting as _dhcp to /var/empty
sandbox: capsicum
spawned master process on PID 73530
spawned privileged actioneer on PID 73531
spawned network proxy on PID 73532
spawned controller proxy on PID 73533
DUID [elided]
bge1: executing: /usr/local/libexec/dhcpcd-run-hooks PREINIT
bge1: executing: /usr/local/libexec/dhcpcd-run-hooks CARRIER
bge1: IAID [elided]
bge1: IA type 3 IAID 00:00:00:01
bge1: IA type 25 IAID 00:00:00:02
bge1: delaying IPv6 router solicitation for 0.0 seconds
bge1: reading lease: /var/db/dhcpcd/bge1.lease6
bge1: rebinding prior DHCPv6 lease
bge1: delaying REBIND6 (xid 0x93b113), next in 1.0 seconds
vlan1: activating for delegation
vlan1: executing: /usr/local/libexec/dhcpcd-run-hooks PREINIT
vlan1: executing: /usr/local/libexec/dhcpcd-run-hooks CARRIER
vlan1: IAID ff:00:03:e9
vlan2: activating for delegation
vlan2: executing: /usr/local/libexec/dhcpcd-run-hooks PREINIT
vlan2: executing: /usr/local/libexec/dhcpcd-run-hooks CARRIER
vlan2: IAID ff:00:03:eb
vlan3: activating for delegation
vlan3: executing: /usr/local/libexec/dhcpcd-run-hooks PREINIT
vlan3: executing: /usr/local/libexec/dhcpcd-run-hooks CARRIER
vlan3: IAID ff:00:00:07
bge1: delaying IPv4 for 1.9 seconds
bge1: soliciting an IPv6 router
bge1: sending Router Solicitation
bge1: Router Advertisement from fe80::[elided]
bge1: adding default route via fe80::[elided]
bge1: executing: /usr/local/libexec/dhcpcd-run-hooks ROUTERADVERT
bge1: broadcasting REBIND6 (xid 0x93b113), next in 1.0 seconds
bge1: REPLY6 received from fe80::[elided]
bge1: adding address 2a02:[elided]/128
bge1: pltime 8046 seconds, vltime 36846 seconds
bge1: renew in 4023, rebind in 6436, expire in 36846 seconds
lo0: adding reject route to 2a02:[elided]::/56 via ::1
bge1: writing lease: /var/db/dhcpcd/bge1.lease6
bge1: delegated prefix 2a02:[elided]::/56
vlan1: adding address 2a02:[elided]1:[elided]/64
vlan1: pltime 8046 seconds, vltime 36846 seconds
vlan1: executing: /usr/local/libexec/dhcpcd-run-hooks DELEGATED6
vlan2: adding address 2a02:[elided]2:[elided]/64
vlan2: pltime 8046 seconds, vltime 36846 seconds
vlan2: executing: /usr/local/libexec/dhcpcd-run-hooks DELEGATED6
vlan3: adding address 2a02:[elided]3:[elided]/64
vlan3: pltime 8046 seconds, vltime 36846 seconds
vlan3: executing: /usr/local/libexec/dhcpcd-run-hooks DELEGATED6
vlan1: adding route to 2a02:[elided]1::/64
vlan2: adding route to 2a02:[elided]2::/64
vlan3: adding route to 2a02:[elided]3::/64
bge1: executing: /usr/local/libexec/dhcpcd-run-hooks REBIND6

I'll build a top of tree checkout and see if I can convince it to produce more logging.

[ppaeps]

the exact text dhcpcd logs when deleting your static routes

Mmhm. You mention static. The kernel-created routes don't have the RTF_STATIC bit. I wonder if this is related to a FreeBSD issue you reported a while ago: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194485.

Unfortunately, setting the RTF_STATIC bit on the routes has no effect, they still get deleted when dhcpcd starts. I.e:

ifconfig vlan1 inet6 fd4a:9620:34ad:1::ff prefixlen 64 -alias
ifconfig vlan1 inet6 fd4a:9620:34ad:1::ff prefixlen 64 alias
ifconfig vlan2 inet6 fd4a:9620:34ad:2::ff prefixlen 64 -alias
ifconfig vlan2 inet6 fd4a:9620:34ad:2::ff prefixlen 64 alias
ifconfig vlan3 inet6 fd4a:9620:34ad:3::ff prefixlen 64 -alias
ifconfig vlan3 inet6 fd4a:9620:34ad:3::ff prefixlen 64 alias
route -6 change -net fd4a:9620:34ad:1::/64 -static
route -6 change -net fd4a:9620:34ad:2::/64 -static
route -6 change -net fd4a:9620:34ad:3::/64 -static
dhcpcd -d

I'll go run that build now to get more debugging output.

[ppaeps]

Unfortunately, a top of tree checkout does not appear produce any more debugging information. I'll try to find some time this evening to dtrace it.

[rsmarples]

So the good news is that it looks like dhcpcd does not appear to be actively deleting your routes. The bad news is that it doesn't help working out what does.

I'm pretty sure that FreeBSD bug has no relation to this issue.

What you could do is run route monitor and check which process deleted the routes. That might help track this down.

[ppaeps]

Oh. So this is very interesting!

For completeness, in case it matters, for these tests I'm running dhcpcd -d after a dhcpcd -x and I have persistent in my configuration file. This particular machine is sitting in an attic about 10,000km from me and I don't have meaningful out of band access to it. The ULA routes are also deleted when dhcpcd starts without previous delegated prefixes configured -- i.e. only the ULA configuration following a reboot.

When I start dhcpcd, I see a series of RTM_DELETE messages:

got message of size 344 on Thu Oct  7 08:53:16 2021
RTM_DELETE: Delete Route: len 344, pid: 0, seq 0, errno 0, flags:<DONE>
locks:  inits:
sockaddrs: <DST,GATEWAY,NETMASK,IFP,IFA>
 2a02:[elided]1:: link#4 (255) ffff ffff ffff ffff ffff ffff ffff vlan1:[elided] fe80::[elided]%vlan1
[...]
got message of size 344 on Thu Oct  7 08:53:16 2021
RTM_DELETE: Delete Route: len 344, pid: 0, seq 0, errno 0, flags:<DONE>
locks:  inits:
sockaddrs: <DST,GATEWAY,NETMASK,IFP,IFA>
 fd4a:9620:34ad:1:: link#6 (255) ffff ffff ffff ffff ffff ffff ffff vlan1:[elided] fe80::[elided]%vlan1

Note pid: 0 here. The kernel is deleting these routes.

Following these, I see a series of messages from pid 2638 (dhcpcd privileged actioneer): default route, reject route, prefix routes. For every prefix route dhcpcd adds, I see the kernel add an interface route:

got message of size 304 on Thu Oct  7 08:53:17 2021
RTM_ADD: Add Route: len 304, pid: 2638, seq 5, errno 17, flags:<UP>
locks:  inits:
sockaddrs: <DST,GATEWAY,NETMASK,IFA>
 2a02:[elided]1:: link#4 ffff:ffff:ffff:ffff:: 2a02:[elided]1:[elided]
got message of size 344 on Thu Oct  7 08:53:17 2021
RTM_ADD: Add Route: len 344, pid: 0, seq 0, errno 0, flags:<UP,DONE>
locks:  inits:
sockaddrs: <DST,GATEWAY,NETMASK,IFP,IFA>
 2a02:[elided]1:: link#4 (255) ffff ffff ffff ffff ffff ffff ffff vlan1:[elided] fe80::[elided]%vlan1

So it's the kernel deleting the routes. And it looks like it's deleting them before dhcpcd installs the default route.

[rsmarples]

I think it's because you're only enabling ipv6rs for the bge0 interface. When dhcpcd does any ipv6rs it performs SIOCSRTRFLUSH_IN6 and SIOCSPFXFLUSH_IN6, passing in the interface name. https://github.com/NetworkConfiguration/dhcpcd/blob/master/src/if-bsd.c#L1895 As you can see by the comment, it's not desirable to stop these ioctls as the kernel would then try to expire stuff dhcpcd is managing.

On FreeBSD at least, I think it ignores the interface and flushes all RA knowledge. You could let dhcpcd manage the RA's on your vlan interfaces to solve this issue, or fix FreeBSD to respect the parameter passed to the ioctls it defines.

[ppaeps]

The ULA prefixes on the VLAN interfaces are manually configured though, they're not learned from router advertisements. But your theory about the SIOCSRTRFLUSH_IN6 and SIOCSPFXFLUSH_IN6 ioctls being the culprits sounds very plausible. On FreeBSD, the kernel does indeed ignore the interface for those ioctls.

As a workaround, I added static ip6_address stanzas for the VLAN interfaces to my dhcpcd.conf:

interface vlan1
  noipv4
  static ip6_address=fd4a:9620:34ad:1::ff/64

The routes are still removed when dhcpcd starts, but this brings them back. For this setup, that is perfectly acceptable. I'll see if I can fix FreeBSD to respect the interface parameter to those ioctls. Meanwhile, I'll also cook up a documentation patch for others who run into this.

Thanks for your help!

[rsmarples]

This is a bug with FreeBSD.