Systemd Networking systemd-networkd.service systemd-networking-wait-online Not Working on/before Clear Linux Build 42170

In the process of updating my system from Clear Linux 42050 to 42170. I experienced network configuration issue. I have been using systemd-networkd.service and iptables for nearly 2 years now. But with this latest update it is failing, iptables is not configuring any rules. I configured my desired ethernet interface routing by using the files in /etc/systemd/network. I have NetworkManager.service turned off I have looked into the problem and it seems that something has changed between Clear Linux build 42050 and 42170 that affects systemd boot timing, as a result network routing comes up broken.

I use systemd-netword.service to configure my network.

ls -l /etc/systemd/network/
total 12
-rw-r–r-- 1 root root 118 Sep 18 2022 10-dhcp.network
-rw-r–r-- 1 root root 756 Oct 8 2022 20-static.network
-rw-r–r-- 1 root root 576 Oct 8 2022 21-static.network

I configure iptable with the file:

/etc/systemd/system/multi-user.target.wants/iptables-restore.service
/etc/iptables.rules

When the system boots this what the status of systemd-networkd.service shows:

● systemd-networkd.service - Network Configuration
     Loaded: loaded (/etc/systemd/system/systemd-networkd.service; disabled; preset: disabled)
     Active: active (running) since Wed 2024-08-21 08:32:52 PDT; 3min 49s ago
TriggeredBy: ● systemd-networkd.socket
       Docs: man:systemd-networkd.service(8)
             man:org.freedesktop.network1(5)
   Main PID: 2172 (systemd-network)
     Status: "Processing requests..."
      Tasks: 1 (limit: 38331)
   FD Store: 0 (limit: 512)
     Memory: 5.0M ()
     CGroup: /system.slice/systemd-networkd.service
             └─2172 /usr/lib/systemd/systemd-networkd

Aug 21 08:32:56 netserver03 systemd-networkd[2172]: enp6s0: Gained carrier
Aug 21 08:32:57 netserver03 systemd-networkd[2172]: enp6s0: Could not enable IP masquerading: Connection timed out
Aug 21 08:32:57 netserver03 systemd-networkd[2172]: enp6s0: Failed
Aug 21 08:32:57 netserver03 systemd-networkd[2172]: enp5s0: Gained IPv6LL
Aug 21 08:32:57 netserver03 systemd-networkd[2172]: enp9s0: Gained carrier
Aug 21 08:32:57 netserver03 systemd-networkd[2172]: enp6s0: Gained IPv6LL
Aug 21 08:32:58 netserver03 systemd-networkd[2172]: enp9s0: DHCPv4 address xx.xx.xx.xxx/23, gateway xx.xx.x.x acquired from xx.xxx.xxx.xx
Aug 21 08:32:58 netserver03 systemd-networkd[2172]: enp9s0: Gained IPv6LL
Aug 21 08:33:01 netserver03 systemd-networkd[2172]: enp9s0: DHCPv6 address ip6 address not shown (valid for 15h 53min 32s, preferred for 15h 53min 32s)
Aug 21 08:33:01 netserver03 systemd-networkd[2172]: enp9s0: DHCP: received delegated prefix 2601:647:8500:ea7::/64

Failed systed-networkd-wait-online.service it did not run

○ systemd-networkd-wait-online.service - Wait for Network to be Configured
Loaded: loaded (/usr/lib/systemd/system/systemd-networkd-wait-online.service; Active: inactive (dead)
Docs: man:systemd-networkd-wait-online.service(8)

The only way to fix the network routing is to run an alias I created after the server has booted in a terminal shell.

alias rs_netw='systemctl restart systemd-networkd.service ; systemctl restart iptables
restore.service ; networkctl '

When systemd-netword.service was restarted manually below is the expected status.

● systemd-networkd.service - Network Configuration
     Loaded: loaded (/etc/systemd/system/systemd-networkd.service; disabled; preset: disabled)
     Active: active (running) since Wed 2024-08-21 08:51:45 PDT; 29min ago
TriggeredBy: ● systemd-networkd.socket
       Docs: man:systemd-networkd.service(8)
             man:org.freedesktop.network1(5)
   Main PID: 3536 (systemd-network)
     Status: "Processing requests..."
      Tasks: 1 (limit: 38331)
   FD Store: 0 (limit: 512)
     Memory: 1.6M ()
     CGroup: /system.slice/systemd-networkd.service
             └─3536 /usr/lib/systemd/systemd-networkd

Aug 21 08:51:45 netserver03 systemd-networkd[3536]: enp5s0: Gained IPv6LL
Aug 21 08:51:45 netserver03 systemd-networkd[3536]: enp6s0: Gained IPv6LL
Aug 21 08:51:45 netserver03 systemd-networkd[3536]: Enumeration completed
Aug 21 08:51:45 netserver03 systemd[1]: Started systemd-networkd.service.
Aug 21 08:51:45 netserver03 systemd-networkd[3536]: enp9s0: Configuring with /etc/systemd/network/10-dhcp.network.
Aug 21 08:51:45 netserver03 systemd-networkd[3536]: enp5s0: Configuring with /etc/systemd/network/20-static.network.
Aug 21 08:51:45 netserver03 systemd-networkd[3536]: enp6s0: Configuring with /etc/systemd/network/21-static.network.
Aug 21 08:51:53 netserver03 systemd-networkd[3536]: enp9s0: DHCPv4 address xx.xxx.x.xxx/23, gateway xx.xx.x.1 acquired from xx.xxx.xxx.xx
Aug 21 08:51:54 netserver03 systemd-networkd[3536]: enp9s0: DHCPv6 address (Valid ipv6 address) (valid for 15h 34min 38s, preferred for 15h 34min 38s)

The networkd-systemd.service defines the conditions for restart if by my understanding, references systemd-networkd-wait-online.service, None of this seems to be working. I have have made changes to systemd-networkd-wait-online to define a valid state for my my network. If I test the unit systemd-networkd-wait-online these changes work by itself. However the changes to systemd-networkd-wait-online did not work when rebooting from a power cycle.

Here is the unit file definition for systemd-netword.service:

 cat /etc/systemd/system/systemd-networkd.service
#  SPDX-License-Identifier: LGPL-2.1-or-later
#
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Network Configuration
Documentation=man:systemd-networkd.service(8)
Documentation=man:org.freedesktop.network1(5)
ConditionCapability=CAP_NET_ADMIN
DefaultDependencies=no
# systemd-udevd.service can be dropped once tuntap is moved to netlink
After=systemd-networkd.socket systemd-udevd.service network-pre.target systemd-sysusers.service systemd-sysctl.service
Before=network.target multi-user.target shutdown.target initrd-switch-root.target
Conflicts=shutdown.target initrd-switch-root.target
Wants=systemd-networkd.socket network.target

[Service]
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_NET_BROADCAST CAP_NET_RAW
BusName=org.freedesktop.network1
CapabilityBoundingSet=CAP_NET_ADMIN CAP_NET_BIND_SERVICE CAP_NET_BROADCAST CAP_NET_RAW
DeviceAllow=char-* rw
ExecStart=!!/usr/lib/systemd/systemd-networkd
FileDescriptorStoreMax=512
LockPersonality=yes
MemoryDenyWriteExecute=yes
NoNewPrivileges=yes
ProtectProc=invisible
ProtectClock=yes
ProtectControlGroups=yes
ProtectHome=yes
ProtectKernelLogs=yes
ProtectKernelModules=yes
ProtectSystem=strict
Restart=on-failure
RestartKillSignal=SIGUSR2
RestartSec=0
RestrictAddressFamilies=AF_UNIX AF_NETLINK AF_INET AF_INET6 AF_PACKET
RestrictNamespaces=yes
RestrictRealtime=yes
RestrictSUIDSGID=yes
RuntimeDirectory=systemd/netif
RuntimeDirectoryPreserve=yes
SystemCallArchitectures=native
SystemCallErrorNumber=EPERM
SystemCallFilter=@system-service
Type=notify-reload
User=systemd-network
WatchdogSec=3min

[Install]
WantedBy=multi-user.target
Also=systemd-networkd.socket
Alias=dbus-org.freedesktop.network1.service

# The output from this generator is used by udevd and networkd. Enable it by
# default when enabling systemd-networkd.service.
Also=systemd-network-generator.service

# We want to enable systemd-networkd-wait-online.service whenever this service
# is enabled. systemd-networkd-wait-online.service has
# WantedBy=network-online.target, so enabling it only has an effect if
# network-online.target itself is enabled or pulled in by some other unit.
Also=systemd-networkd-wait-online.service

Other details my server hardware and relevant software versions:

lshw -class cpu -short;lshw -class display;lshw -class memory -short;
H/W path                    Device          Class       Description
===================================================================
/0/37                                       processor   AMD Ryzen 9 5950X 16-Core Processor
  *-display
       description: VGA compatible controller
       product: DG2 [Arc A770]
       vendor: Intel Corporation
       physical id: 0
       bus info: pci@0000:0c:00.0
       version: 08
       width: 64 bits
       clock: 33MHz
       capabilities: pciexpress msi pm vga_controller bus_master cap_list rom
       configuration: driver=i915 latency=0
       resources: irq:43 memory:fb000000-fbffffff memory:d0000000-dfffffff memory:c0000-dffff
H/W path                    Device          Class       Description
===================================================================
/0/0                                        memory      64KiB BIOS
/0/31                                       memory      32GiB System Memory
/0/31/0                                     memory      16GiB DIMM DDR4 Synchronous Unbuffered (Unregistered) 2400 MHz (0.4 ns)
/0/31/1                                     memory      [empty]
/0/31/2                                     memory      16GiB DIMM DDR4 Synchronous Unbuffered (Unregistered) 2400 MHz (0.4 ns)
/0/31/3                                     memory      [empty]
/0/34                                       memory      1MiB L1 cache
/0/35                                       memory      8MiB L2 cache
/0/36                                       memory      64MiB L3 cache
#
#
#
swupd info ; systemctl --version ; uname -a
Distribution:      Clear Linux OS
Installed version: 42170
Version URL:       https://cdn.download.clearlinux.org/update
Content URL:       https://cdn.download.clearlinux.org/update
systemd 255 (255)
+PAM +AUDIT -SELINUX -APPARMOR +IMA -SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 -IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK -PCRE2 +PWQUALITY +P11KIT -QRENCODE +TPM2 +BZIP2 -LZ4 -XZ +ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP -SYSVINIT default-hierarchy=hybrid
Linux netserver03 6.10.5-1458.native #1 SMP Wed Aug 14 07:09:01 PDT 2024 x86_64 GNU/Linux

clearlinux / distribution

Systemd Networking systemd-networkd.service systemd-networking-wait-online Not Working on/before Clear Linux Build 42170 #3173