systemd-networkd fails when using physical(!) Incus network

To be honest, I'm not sure where to file this bug... lxc, Incus, or systemd? Would love to hear your thoughts on the issue. For now I have a usable workaround (using ifupdown, details below), but it would be helpful if this bug would be fixed, so everybody can benefit from things working out-of-the-box.

EDIT: I probably should've filed this under lxc/incus instead of zabbly/incus. Sorry! EDIT2: Incus version: 6.0.1 LTS

Background:

I'm running an Incus host in QEMU/KVM. Hence, I've added multiple virtual NICs to the Incus host, so I can give each container its own NIC, in order to avoid running into issues related to things like 'port security' and multiple MAC addresses originating from the same NIC (as would be the case with macvlan).

On the Incus host:

incus network create testphys0 -t physical parent=enp9s0

Then I assigned the physical 'testphys0' Incus network to a regular Debian 12 (bookworm) system container.

Or, in one step (ignoring the above):

incus config device add testcontainer eth0 nic nictype=physical parent=enp9s0

The problem:

In the container:

The systemd-networkd daemon is NOT assigning an IP address to the eth0 interface. I've tried using a static IP as well as DHCP. There's no difference in symptoms. The eth0 interface is UP but is NOT being assigned an IP address.

The networkctl command shows the following:

  IDX LINK TYPE     OPERATIONAL SETUP
    1 lo   loopback carrier     unmanaged
    6 eth0 ether    degraded    pending

  2 links listed.

The ip -br a command shows:

  lo               UNKNOWN        127.0.0.1/8 ::1/128
  eth0             UP             fe80::5054:ff:fecc:ef0a/64

(Disabling IPv6 doesn't matter either.)

I've also included the systemd-networkd logs (with debugging output). See further below.

A dirty, undesirable workaround is to run the container in privileged mode, but for security reasons that's NOT a viable solution. For those interested in a better (safer) workaround, see below.

Now the question:

Is this a bug in systemd-networkd or somewhere else?

Workaround (or solution), for others reading this:

This is a workaround which does NOT require the container to be in 'privileged mode':

This will disable systemd-networkd and it will remove the systemd-resolved package. The systemd-resolved package is removed by installing resolvconf (which we will need if we want to configure DNS settings via the ifupdown interfaces config file).

  $ systemctl disable --now systemd-networkd.service
  $ systemctl disable --now systemd-networkd.socket
  $ apt install ifupdown resolvconf
  $ cat <<EOF > /etc/network/interfaces.d/eth0
  auto eth0
  iface eth0 inet static
          address 192.168.122.26/24
          gateway 192.168.122.1
          # Space-separated list of DNS servers.
          dns-nameservers 192.168.122.1
          # Search suffix for DNS lookups having fewer
          # than 'ndots' dots (default 1).
          dns-search example.localdomain
  EOF

Note, it seems you must use 'auto eth0' and NOT 'allow-hotplug eth0', otherwise it won't work. That might be because ifupdown's networking scripts won't actually be triggered by the hotplug (udev) event because the interface is being renamed when the container is starting up, meaning the hotplug event was missed (the interface had a different name at the moment of the udev event).

Other debugging details:

If we give the container a bridged Incus network, like 'incusbr0', there's no issue at all. The problem only occurs if we assign a physical Incus network to the container, as shown at the beginning of this post.
I've read lxc issue #2278 and that made me check AppArmor. For completeness sake: installing apparmor and running 'aa-teardown' doesn't make a difference; if we subsequently run 'systemctl restart systemd-networkd', the symptoms stay the same.
Just to rule it out, I tried changing the Incus VM's virtual NIC driver from virtio to e1000e, without any noticeable difference.

Logs:

Logs from the (non-privileged) container, with SYSTEMD_LOG_LEVEL=debug enabled:

# journalctl -eu systemd-networkd

Jul 09 19:50:05 example systemd[1]: Stopping systemd-networkd.service - Network Configuration...
Jul 09 19:50:05 example systemd[1]: systemd-networkd.service: Deactivated successfully.
Jul 09 19:50:05 example systemd[1]: Stopped systemd-networkd.service - Network Configuration.
Jul 09 19:50:05 example systemd[1]: Starting systemd-networkd.service - Network Configuration...
Jul 09 19:50:05 example systemd-networkd[523]: SELinux enabled state cached to: disabled
Jul 09 19:50:05 example systemd-networkd[523]: Failed to increase receive buffer size for general netlink socket, ignoring: Operation not permitted
Jul 09 19:50:05 example systemd-networkd[523]: Bus bus-api-network: changing state UNSET → OPENING
Jul 09 19:50:05 example systemd-networkd[523]: sd-bus: starting bus bus-api-network by connecting to /run/dbus/system_bus_socket...
Jul 09 19:50:05 example systemd-networkd[523]: Bus bus-api-network: changing state OPENING → AUTHENTICATING
Jul 09 19:50:05 example systemd-networkd[523]: Registering bus object implementation for path=/org/freedesktop/network1 iface=org.freedesktop.network1.Manager
Jul 09 19:50:05 example systemd-networkd[523]: Registering bus object implementation for path=/org/freedesktop/network1/link iface=org.freedesktop.network1.DHCPServer
Jul 09 19:50:05 example systemd-networkd[523]: Registering bus object implementation for path=/org/freedesktop/network1/link iface=org.freedesktop.network1.Link
Jul 09 19:50:05 example systemd-networkd[523]: Registering bus object implementation for path=/org/freedesktop/network1/network iface=org.freedesktop.network1.Network
Jul 09 19:50:05 example systemd-networkd[523]: Registering bus object implementation for path=/org/freedesktop/LogControl1 iface=org.freedesktop.LogControl1
Jul 09 19:50:05 example systemd-networkd[523]: sd-device-monitor(n/a): No permission to stat PID1's netns, unable to determine if we are in host netns, ignoring: Permission denied
Jul 09 19:50:05 example systemd-networkd[523]: sd-device-monitor(n/a): Failed to increase receive buffer size, ignoring: Operation not permitted
Jul 09 19:50:05 example systemd-networkd[523]: eth0: Saved new link: ifindex=6, iftype=ETHER(1), kind=n/a
Jul 09 19:50:05 example systemd-networkd[523]: eth0: Saved original MTU 1500 (min: 68, max: 65535)
Jul 09 19:50:05 example systemd-networkd[523]: eth0: Saved hardware address: 52:54:00:cc:ef:0a
Jul 09 19:50:05 example systemd-networkd[523]: eth0: Saved IPv6 link-local address generation mode: eui64
Jul 09 19:50:05 example systemd-networkd[523]: eth0: Flags change: +UP +LOWER_UP +RUNNING +MULTICAST +BROADCAST
Jul 09 19:50:05 example systemd-networkd[523]: eth0: Link UP
Jul 09 19:50:05 example systemd-networkd[523]: eth0: Gained carrier
Jul 09 19:50:05 example systemd-networkd[523]: eth0: link pending udev initialization...
Jul 09 19:50:05 example systemd-networkd[523]: lo: Saved new link: ifindex=1, iftype=LOOPBACK(772), kind=n/a
Jul 09 19:50:05 example systemd-networkd[523]: lo: Saved original MTU 65536 (min: 0, max: 4294967295)
Jul 09 19:50:05 example systemd-networkd[523]: lo: Saved hardware address: 00:00:00:00:00:00
Jul 09 19:50:05 example systemd-networkd[523]: lo: Saved IPv6 link-local address generation mode: eui64
Jul 09 19:50:05 example systemd-networkd[523]: lo: Flags change: +LOOPBACK +UP +LOWER_UP +RUNNING
Jul 09 19:50:05 example systemd-networkd[523]: lo: Link UP
Jul 09 19:50:05 example systemd-networkd[523]: lo: Gained carrier
Jul 09 19:50:05 example systemd-networkd[523]: lo: udev initialized link
Jul 09 19:50:05 example systemd-networkd[523]: lo: State changed: pending -> initialized
Jul 09 19:50:05 example systemd-networkd[523]: eth0: Received new foreign QDisc (configured): handle=0:0, parent=ffff:ffff, kind=fq_codel
Jul 09 19:50:05 example systemd-networkd[523]: lo: Received new foreign QDisc (configured): handle=0:0, parent=ffff:ffff, kind=noqueue
Jul 09 19:50:05 example systemd-networkd[523]: eth0: Received new foreign address (configured): fe80::5054:ff:fecc:ef0a/64 (valid forever, preferred forever), flags: permanent, scope: link
Jul 09 19:50:05 example systemd-networkd[523]: eth0: Gained IPv6LL
Jul 09 19:50:05 example systemd-networkd[523]: eth0: link_check_ready(): link is in pending state.
Jul 09 19:50:05 example systemd-networkd[523]: lo: Received new foreign address (configured): ::1/128 (valid forever, preferred forever), flags: permanent, scope: host
Jul 09 19:50:05 example systemd-networkd[523]: lo: link_check_ready(): link is in initialized state.
Jul 09 19:50:05 example systemd-networkd[523]: lo: Received new foreign address (configured): 127.0.0.1/8 (valid forever, preferred forever), flags: permanent, scope: host
Jul 09 19:50:05 example systemd-networkd[523]: lo: link_check_ready(): link is in initialized state.
Jul 09 19:50:05 example systemd-networkd[523]: rtnl: received non-static neighbor, ignoring.
Jul 09 19:50:05 example systemd-networkd[523]: rtnl: received non-static neighbor, ignoring.
Jul 09 19:50:05 example systemd-networkd[523]: rtnl: received non-static neighbor, ignoring.
Jul 09 19:50:05 example systemd-networkd[523]: rtnl: received non-static neighbor, ignoring.
Jul 09 19:50:05 example systemd-networkd[523]: rtnl: received non-static neighbor, ignoring.
Jul 09 19:50:05 example systemd-networkd[523]: eth0: Received new foreign route (configured): dst: ff00::/8, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: local(255), proto: kernel, type: multicast, nexthop: 0, priority: 256, flags: n/a
Jul 09 19:50:05 example systemd-networkd[523]: eth0: Received new foreign route (configured): dst: fe80::5054:ff:fecc:ef0a/128, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: local(255), proto: kernel, type: local, nexthop: 0, priority: 0, flags: n/a
Jul 09 19:50:05 example systemd-networkd[523]: lo: Received new foreign route (configured): dst: ::1/128, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: local(255), proto: kernel, type: local, nexthop: 0, priority: 0, flags: n/a
Jul 09 19:50:05 example systemd-networkd[523]: eth0: Received new foreign route (configured): dst: fe80::/64, src: n/a, gw: n/a, prefsrc: n/a, scope: global, table: main(254), proto: kernel, type: unicast, nexthop: 0, priority: 256, flags: n/a
Jul 09 19:50:05 example systemd-networkd[523]: lo: Received new foreign route (configured): dst: 127.255.255.255/32, src: n/a, gw: n/a, prefsrc: 127.0.0.1, scope: link, table: local(255), proto: kernel, type: broadcast, nexthop: 0, priority: 0, flags: n/a
Jul 09 19:50:05 example systemd-networkd[523]: lo: Received new foreign route (configured): dst: 127.0.0.1/32, src: n/a, gw: n/a, prefsrc: 127.0.0.1, scope: host, table: local(255), proto: kernel, type: local, nexthop: 0, priority: 0, flags: n/a
Jul 09 19:50:05 example systemd-networkd[523]: lo: Received new foreign route (configured): dst: 127.0.0.0/8, src: n/a, gw: n/a, prefsrc: 127.0.0.1, scope: host, table: local(255), proto: kernel, type: local, nexthop: 0, priority: 0, flags: n/a
Jul 09 19:50:05 example systemd-networkd[523]: Remembering foreign routing policy rule (configured): priority: 32766, ::/0 -> ::/0, iif: n/a, oif: n/a, table: main(254)
Jul 09 19:50:05 example systemd-networkd[523]: Remembering foreign routing policy rule (configured): priority: 0, ::/0 -> ::/0, iif: n/a, oif: n/a, table: local(255)
Jul 09 19:50:05 example systemd-networkd[523]: rtnl: received rule message with invalid family 129, ignoring.
Jul 09 19:50:05 example systemd-networkd[523]: rtnl: received rule message with invalid family 128, ignoring.
Jul 09 19:50:05 example systemd-networkd[523]: Remembering foreign routing policy rule (configured): priority: 32767, 0.0.0.0/0 -> 0.0.0.0/0, iif: n/a, oif: n/a, table: default(253)
Jul 09 19:50:05 example systemd-networkd[523]: Remembering foreign routing policy rule (configured): priority: 32766, 0.0.0.0/0 -> 0.0.0.0/0, iif: n/a, oif: n/a, table: main(254)
Jul 09 19:50:05 example systemd-networkd[523]: Remembering foreign routing policy rule (configured): priority: 0, 0.0.0.0/0 -> 0.0.0.0/0, iif: n/a, oif: n/a, table: local(255)
Jul 09 19:50:05 example systemd-networkd[523]: Enumeration completed
Jul 09 19:50:05 example systemd[1]: Started systemd-networkd.service - Network Configuration.
Jul 09 19:50:05 example systemd-networkd[523]: Bus bus-api-network: changing state AUTHENTICATING → HELLO
Jul 09 19:50:05 example systemd-networkd[523]: Sent message type=method_call sender=n/a destination=org.freedesktop.DBus path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=Hello cookie=1 reply_cookie=0 signature=n/a error-name=n/a error-message=n/a
Jul 09 19:50:05 example systemd-networkd[523]: Sent message type=method_call sender=n/a destination=org.freedesktop.DBus path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=RequestName cookie=2 reply_cookie=0 signature=su error-name=n/a error-message=n/a
Jul 09 19:50:05 example systemd-networkd[523]: Sent message type=method_call sender=n/a destination=org.freedesktop.DBus path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=AddMatch cookie=3 reply_cookie=0 signature=s error-name=n/a error-message=n/a
Jul 09 19:50:05 example systemd-networkd[523]: Got message type=method_return sender=org.freedesktop.DBus destination=:1.17 path=n/a interface=n/a member=n/a cookie=1 reply_cookie=1 signature=s error-name=n/a error-message=n/a
Jul 09 19:50:05 example systemd-networkd[523]: Bus bus-api-network: changing state HELLO → RUNNING
Jul 09 19:50:05 example systemd-networkd[523]: lo: Failed to get driver, continuing without: Operation not supported
Jul 09 19:50:05 example systemd-networkd[523]: lo: Saved permanent hardware address: 00:00:00:00:00:00
Jul 09 19:50:05 example systemd-networkd[523]: lo: Link state is up-to-date
Jul 09 19:50:05 example systemd-networkd[523]: lo: Unmanaging interface.
Jul 09 19:50:05 example systemd-networkd[523]: lo: State changed: initialized -> unmanaged
Jul 09 19:50:05 example systemd-networkd[523]: Sent message type=signal sender=n/a destination=n/a path=/org/freedesktop/network1/link/_31 interface=org.freedesktop.DBus.Properties member=PropertiesChanged cookie=4 reply_cookie=0 signature=sa{sv}as error-name=n/a error-message=n/a
Jul 09 19:50:05 example systemd-networkd[523]: Got message type=signal sender=org.freedesktop.DBus.Local destination=n/a path=/org/freedesktop/DBus/Local interface=org.freedesktop.DBus.Local member=Connected cookie=4294967295 reply_cookie=0 signature=n/a error-name=n/a error-message=n/a
Jul 09 19:50:05 example systemd-networkd[523]: Got message type=signal sender=org.freedesktop.DBus destination=:1.17 path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=NameAcquired cookie=2 reply_cookie=0 signature=s error-name=n/a error-message=n/a
Jul 09 19:50:05 example systemd-networkd[523]: Got message type=signal sender=org.freedesktop.DBus destination=:1.17 path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=NameAcquired cookie=3 reply_cookie=0 signature=s error-name=n/a error-message=n/a
Jul 09 19:50:05 example systemd-networkd[523]: Got message type=method_return sender=org.freedesktop.DBus destination=:1.17 path=n/a interface=n/a member=n/a cookie=4 reply_cookie=2 signature=u error-name=n/a error-message=n/a
Jul 09 19:50:05 example systemd-networkd[523]: Successfully acquired requested service name.
Jul 09 19:50:05 example systemd-networkd[523]: Got message type=method_return sender=org.freedesktop.DBus destination=:1.17 path=n/a interface=n/a member=n/a cookie=5 reply_cookie=3 signature=n/a error-name=n/a error-message=n/a
Jul 09 19:50:05 example systemd-networkd[523]: Match type='signal',sender='org.freedesktop.login1',path='/org/freedesktop/login1',interface='org.freedesktop.login1.Manager',member='PrepareForSleep' successfully installed.

PS:

You've been doing good work Stéphane! Much love :heart:

zabbly / incus