search

lueschem / edi-cl

This repository contains an edi project configuration for Compulab embedded devices.
https://www.get-edi.io
Other
10 stars 2 forks source link

Switch Digital Twin Container to NetworkManager #19

Open lueschem opened 3 years ago

lueschem commented 3 years ago

In order to keep the digital twin LXD container as close as possible to the real hardware it would make sense to switch the network management from ifup to NetworkManager also within the LXD container.

For this the package network-manager needs to be installed during bootstrapping. After this there is no need to separately configure eth0 as the default NetworkManager config is DHCP for wired network interfaces.

The switch to NetworkManager within the LXD container could lead to temporary incompatibilities and special care needs to be taken if an existing LXD container gets switched over to NetworkManager. Maybe the container should get re-created from scratch or as a new configuration.

lueschem commented 3 years ago

On bullseye there seems to be a problem with NetworkManager:

Jan 08 10:52:50 var-som-mx8m-nano-bullseye-arm64-cross-dev systemd[1]: Starting Network Manager...
Jan 08 10:52:50 var-som-mx8m-nano-bullseye-arm64-cross-dev systemd[372]: NetworkManager.service: Failed to set up mount namespacing: Permission denied
Jan 08 10:52:50 var-som-mx8m-nano-bullseye-arm64-cross-dev systemd[372]: NetworkManager.service: Failed at step NAMESPACE spawning /usr/sbin/NetworkManager: Permission denied
Jan 08 10:52:50 var-som-mx8m-nano-bullseye-arm64-cross-dev systemd[1]: NetworkManager.service: Main process exited, code=exited, status=226/NAMESPACE
Jan 08 10:52:50 var-som-mx8m-nano-bullseye-arm64-cross-dev systemd[1]: NetworkManager.service: Failed with result 'exit-code'.
Jan 08 10:52:50 var-som-mx8m-nano-bullseye-arm64-cross-dev systemd[1]: Failed to start Network Manager.
Jan 08 10:52:50 var-som-mx8m-nano-bullseye-arm64-cross-dev systemd[1]: NetworkManager.service: Scheduled restart job, restart counter is at 5.
Jan 08 10:52:50 var-som-mx8m-nano-bullseye-arm64-cross-dev systemd[1]: Stopped Network Manager.
Jan 08 10:52:50 var-som-mx8m-nano-bullseye-arm64-cross-dev systemd[1]: NetworkManager.service: Start request repeated too quickly.
Jan 08 10:52:50 var-som-mx8m-nano-bullseye-arm64-cross-dev systemd[1]: NetworkManager.service: Failed with result 'exit-code'.
Jan 08 10:52:50 var-som-mx8m-nano-bullseye-arm64-cross-dev systemd[1]: Failed to start Network Manager.
lueschem commented 3 years ago

lxc config set container security.nesting true allows NetworkManager to start properly.

lueschem commented 3 years ago

It looks like it is not straightforward to get NetworkManager running within a LXD container. Here are some first observations:

Within a standard Debian bullseye container eth0 (IPv4, DHCP) comes up properly thanks to ifupdown:

me@notebook:~$ lxc launch images:debian/11/amd64 debian-bullseye-nm
me@notebook:~$ lxc exec debian-bullseye-nm bash
root@debian-bullseye-nm:~# ip addr show eth0
36: eth0@if37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:d4:da:c4 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.215.5.230/24 brd 10.215.5.255 scope global dynamic eth0
       valid_lft 3501sec preferred_lft 3501sec
    inet6 fd42:53b4:a820:18f7:216:3eff:fed4:dac4/64 scope global dynamic mngtmpaddr 
       valid_lft 3576sec preferred_lft 3576sec
    inet6 fe80::216:3eff:fed4:dac4/64 scope link 
       valid_lft forever preferred_lft forever

Now we switch to NetworkManager:

root@debian-bullseye-nm:~# apt install network-manager
root@debian-bullseye-nm:~# apt purge ifupdown
root@debian-bullseye-nm:~# rm /etc/network/interfaces
root@debian-bullseye-nm:~# reboot
me@notebook:~$ lxc exec debian-bullseye-nm bash
root@debian-bullseye-nm:~# nmcli device
DEVICE  TYPE      STATE      CONNECTION 
eth0    ethernet  unmanaged  --         
lo      loopback  unmanaged  --

Result: Within an unprivileged container NetworkManager decides to not manager eth0 (strictly unmanaged). The root cause for this behavior yet needs to be figured out.

As a result eth0 does not get an IPv4 address via DHCP:

root@debian-bullseye-nm:~# ifconfig -a
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::216:3eff:fed4:dac4  prefixlen 64  scopeid 0x20<link>
        inet6 fd42:53b4:a820:18f7:216:3eff:fed4:dac4  prefixlen 64  scopeid 0x0<global>
        ether 00:16:3e:d4:da:c4  txqueuelen 1000  (Ethernet)
        RX packets 24  bytes 3526 (3.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 9  bytes 766 (766.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Now we switch the container into privileged mode

root@debian-bullseye-nm:~# exit
me@notebook:~$ lxc config set debian-bullseye-nm security.privileged true
me@notebook:~$ lxc stop debian-bullseye-nm
me@notebook:~$ lxc stop debian-bullseye-nm
me@notebook:~$ lxc exec debian-bullseye-nm bash

root@debian-bullseye-nm:~# systemctl status NetworkManager
● NetworkManager.service - Network Manager
     Loaded: loaded (/lib/systemd/system/NetworkManager.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2021-01-29 10:03:54 UTC; 1min 0s ago
       Docs: man:NetworkManager(8)
    Process: 82 ExecStart=/usr/sbin/NetworkManager --no-daemon (code=exited, status=226/NAMESPACE)
   Main PID: 82 (code=exited, status=226/NAMESPACE)

Jan 29 10:03:54 debian-bullseye-nm systemd[1]: NetworkManager.service: Scheduled restart job, restart counter is at 5.
Jan 29 10:03:54 debian-bullseye-nm systemd[1]: Stopped Network Manager.
Jan 29 10:03:54 debian-bullseye-nm systemd[1]: NetworkManager.service: Start request repeated too quickly.
Jan 29 10:03:54 debian-bullseye-nm systemd[1]: NetworkManager.service: Failed with result 'exit-code'.
Jan 29 10:03:54 debian-bullseye-nm systemd[1]: Failed to start Network Manager.

Result: In the privileged container NetworkManager does not start.

It looks like it gets blocked by Apparmor:

root@debian-bullseye-nm:~# exit
me@notebook:~$ dmesg
...
[ 8042.785535] audit: type=1400 audit(1611914633.415:374): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxd-debian-bullseye-nm_</var/snap/lxd/common/lxd>" name="/run/systemd/unit-root/" pid=59794 comm="(d-logind)" srcname="/" flags="rw, rbind"

Now we enable nesting on the container:

me@notebook:~$ lxc config set debian-bullseye-nm security.nesting true
me@notebook:~$ lxc stop debian-bullseye-nm
me@notebook:~$ lxc stop debian-bullseye-nm
me@notebook:~$ lxc exec debian-bullseye-nm bash
root@debian-bullseye-nm:~# nmcli con show
NAME                UUID                                  TYPE      DEVICE 
Wired connection 1  a199655c-e7e1-370d-ac84-1ecfc2ea2710  ethernet  eth0

Finally NetworkManager manages eth0. But we are not there yet. Sometimes we have "two" eth0 connections due to a race condition. This race condition can be forced like this:

Within the container we delay the NetworkManager daemon startup a bit:

root@debian-bullseye-nm:~# vi /lib/systemd/system/NetworkManager.service
--> add:
ExecStartPre=/bin/sleep 5

Now we reboot the container:

root@debian-bullseye-nm:~# reboot
me@notebook:~$ lxc exec debian-bullseye-nm bash
root@debian-bullseye-nm:~# nmcli con show
NAME                UUID                                  TYPE      DEVICE 
eth0                08ac7954-d6f7-4876-936f-f06122ca62e3  ethernet  eth0   
Wired connection 1  a199655c-e7e1-370d-ac84-1ecfc2ea2710  ethernet  --

Result: The lxdbr0 assigned an IPv6 address to eth0 and therefore NetworkManager does not bring up eth0 automatically anymore.

Now we switch off this behavior on lxdbr0:

root@debian-bullseye-nm:~# exit
me@notebook:~$ lxc network set lxdbr0 ipv6.dhcp.stateful=true
me@notebook:~$ lxc stop debian-bullseye-nm
me@notebook:~$ lxc stop debian-bullseye-nm
me@notebook:~$ lxc exec debian-bullseye-nm bash
root@debian-bullseye-nm:~# nmcli con show
NAME                UUID                                  TYPE      DEVICE 
Wired connection 1  a199655c-e7e1-370d-ac84-1ecfc2ea2710  ethernet  eth0

root@debian-bullseye-nm:~# ip addr show eth0
82: eth0@if83: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:57:91:da brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.215.5.247/24 brd 10.215.5.255 scope global dynamic noprefixroute eth0
       valid_lft 3551sec preferred_lft 3551sec
    inet6 fd42:53b4:a820:18f7:a38b:435:88f0:3760/128 scope global dynamic noprefixroute 
       valid_lft 3553sec preferred_lft 3553sec
    inet6 fe80::9d3:b2fa:4fe5:cc9b/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

Result: After all this modifications NetworkManager works as expected within the LXD container. However, it would be desirable to run the container in unprivileged mode and without nesting enabled.

Bonus:

Check device passthrough:

me@notebook:~$ lxc config device add debian-bullseye-nm eth1 nic nictype=physical parent=enxd46e0e06171b name=eth1
lueschem commented 3 years ago

@thom311: Any help or hints are highly appreciated! It would be really great to get NetworkManager running smoothly within an LXD container.

thom311 commented 3 years ago

in the unpriveledged container, is the problem that the device appears unmanaged?

that might be because NetworkManager thinks udev should be running, but it is not actually running. In that case, udev never reports the interface as initialized, and NetworkManager does not manage the device. NetworkManager decides whether udev is running based on whether /sys is writable.

What is the actual issue?

Most important when debugging NetworkManager is collecting level=TRACE logs. See https://cgit.freedesktop.org/NetworkManager/NetworkManager/tree/contrib/fedora/rpm/NetworkManager.conf#n28 for hints about logging?

lueschem commented 3 years ago

Thanks a lot for the hints!

The final goal would be indeed to get the eth0 managed within the unprivileged container. Actually udev is running within the LXD container:

root@debian-bullseye-nm:~# systemctl status systemd-udevd
● systemd-udevd.service - Rule-based Manager for Device Events and Files
     Loaded: loaded (/lib/systemd/system/systemd-udevd.service; static)
     Active: active (running) since Tue 2021-02-16 11:27:10 UTC; 12s ago
TriggeredBy: ● systemd-udevd-kernel.socket
             ● systemd-udevd-control.socket
       Docs: man:systemd-udevd.service(8)
             man:udev(7)
   Main PID: 120 (systemd-udevd)
     Status: "Processing with 24 children at max"
      Tasks: 1
     Memory: 1.4M
     CGroup: /system.slice/systemd-udevd.service
             └─120 /lib/systemd/systemd-udevd

Feb 16 11:27:10 debian-bullseye-nm systemd[1]: Starting Rule-based Manager for Device Events and Files...
Feb 16 11:27:10 debian-bullseye-nm systemd-udevd[120]: Failed to chown '/dev/net/tun' 0 0: Operation not permitted
Feb 16 11:27:10 debian-bullseye-nm systemd-udevd[120]: Failed to apply permissions on static device nodes: Operation not permitted
Feb 16 11:27:10 debian-bullseye-nm systemd[1]: Started Rule-based Manager for Device Events and Files.

Also the sysfs should be mounted as rw:

root@debian-bullseye-nm:~# mount | grep sysfs
sysfs on /sys type sysfs (rw,relatime)

But probably not with the expected ownership:

root@debian-bullseye-nm:~# ls -n /sys
total 0
drwxr-xr-x   2 65534 65534 0 Feb 16 11:30 block
drwxr-xr-x  50 65534 65534 0 Feb 16 11:25 bus
drwxr-xr-x  79 65534 65534 0 Feb 16 11:25 class
drwxr-xr-x   4 65534 65534 0 Feb 16 11:30 dev
drwxr-xr-x  25 65534 65534 0 Feb 16 11:25 devices
drwxr-xr-x   6 65534 65534 0 Feb 16 11:25 firmware
drwxr-xr-x  10 65534 65534 0 Feb 16 11:25 fs
drwxr-xr-x   2 65534 65534 0 Feb 16 11:30 hypervisor
drwxr-xr-x  14 65534 65534 0 Feb 16 11:25 kernel
drwxr-xr-x 330 65534 65534 0 Feb 16 11:25 module
drwxr-xr-x   3 65534 65534 0 Feb 16 11:30 power

Here is the trace output of NetworkManager (I will further analyze it as soon as possible):

Feb 16 11:40:25 debian-bullseye-nm systemd[1]: Starting Network Manager...
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2434] monotonic timestamp started counting 1.433256697 seconds ago with an offset of 1079.0 seconds to CLOCK_BOOTTIME (local time is 2021-02-16 11:40:25)
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <info>  [1613475625.2435] NetworkManager (version 1.28.0) is starting... (after a restart)
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <info>  [1613475625.2435] Read config: /etc/NetworkManager/NetworkManager.conf (lib: no-mac-addr-change.conf)
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2435] CONFIG: config-data[0x56471ce19190]: 3 groups
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2435] CONFIG: 
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2435] CONFIG: [main]
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2435] CONFIG:   level=TRACE
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2436] CONFIG:   plugins=ifupdown,keyfile
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2436] CONFIG: 
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2436] CONFIG: [ifupdown]
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2436] CONFIG:   managed=false
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2436] CONFIG: 
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2436] CONFIG: [device-31-mac-addr-change]
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2440] CONFIG:   match-device=driver:eagle_sdio,driver:wl
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2441] CONFIG:   wifi.scan-rand-mac-address=no
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2441] CONFIG: 
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2442] CONFIG: # no-auto-default file "/var/lib/NetworkManager/no-auto-default.state"
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <warn>  [1613475625.2442] config: unknown key 'level' in section [main] of file '/etc/NetworkManager/NetworkManager.conf'
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2443] config: state: successfully read state file "/var/lib/NetworkManager/NetworkManager.state"
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2444] WEXT support is enabled
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2444] create NMDBusManager singleton (072109794cb4a407)
Feb 16 11:40:25 debian-bullseye-nm systemd[1]: Started Network Manager.
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <info>  [1613475625.2525] bus-manager: acquired D-Bus service "org.freedesktop.NetworkManager"
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2527] netns[0x56471ce2d720]: new netns (net:8, mnt:9)
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2587] platform-linux: create (ignore netns, initial netns, use udev)
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2590] platform-linux: Netlink socket for events established: port=168, fd=12
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2591] platform-linux: populate platform cache
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2593] platform: kernel-support for USER_IPV6LL (IFLA_INET6_ADDR_GEN_MODE support) detected: supported
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <trace> [1613475625.2594] platform-linux: event-notification: RTM_NEWLINK, flags multi, seq 1, in-dump: 1: lo <UP,LOWER_UP;loopback,up,running,lowerup> mtu 65536 arp 772 loopback? not-init addrgenmode eui64 addr 00:00:00:00:00:00 brd 00:00:00:00:00:00 rx:0,0 tx:0,0
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <trace> [1613475625.2596] ethtool[1]: ETHTOOL_GDRVINFO, lo: failed: Operation not supported
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2598] platform: (lo) signal: link   added: 1: lo <UP,LOWER_UP;loopback,up,running,lowerup> mtu 65536 arp 772 loopback? not-init addrgenmode eui64 addr 00:00:00:00:00:00 brd 00:00:00:00:00:00 driver unknown rx:0,0 tx:0,0
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <trace> [1613475625.2599] platform-linux: event-notification: RTM_NEWLINK, flags multi, seq 1, in-dump: 43: eth0@other-netns <UP,LOWER_UP;broadcast,multicast,up,running,lowerup> mtu 1500 arp 1 veth* not-init addrgenmode eui64 addr 00:16:3E:C2:CE:FA brd FF:FF:FF:FF:FF:FF rx:50,8590 tx:11,950
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2600] platform: (eth0) signal: link   added: 43: eth0@other-netns <UP,LOWER_UP;broadcast,multicast,up,running,lowerup> mtu 1500 arp 1 veth* not-init addrgenmode eui64 addr 00:16:3E:C2:CE:FA brd FF:FF:FF:FF:FF:FF driver veth rx:50,8590 tx:11,950
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2601] platform: kernel-support for EXTENDED_IFA_FLAGS (IPv6 temporary addresses support) detected: supported
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <trace> [1613475625.2602] platform-linux: event-notification: RTM_NEWADDR, flags multi, seq 2, in-dump: 127.0.0.1/8 lft forever pref forever lifetime 1-0[4294967295,4294967295] dev 1 flags permanent src kernel
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2603] platform: (lo) signal: address 4   added: 127.0.0.1/8 lft forever pref forever lifetime 1-0[4294967295,4294967295] dev 1 flags permanent src kernel
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <trace> [1613475625.2605] platform-linux: event-notification: RTM_NEWADDR, flags multi, seq 3, in-dump: ::1/128 lft forever pref forever lifetime 1-0[4294967295,4294967295] dev 1 flags permanent src kernel
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2606] platform: (lo) signal: address 6   added: ::1/128 lft forever pref forever lifetime 1-0[4294967295,4294967295] dev 1 flags permanent src kernel
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <trace> [1613475625.2606] platform-linux: event-notification: RTM_NEWADDR, flags multi, seq 3, in-dump: fe80::216:3eff:fec2:cefa/64 lft forever pref forever lifetime 1-0[4294967295,4294967295] dev 43 flags permanent src kernel
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2606] platform: (eth0) signal: address 6   added: fe80::216:3eff:fec2:cefa/64 lft forever pref forever lifetime 1-0[4294967295,4294967295] dev 43 flags permanent src kernel
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <trace> [1613475625.2607] platform-linux: event-notification: RTM_NEWROUTE, flags multi, seq 4: ignore
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <trace> [1613475625.2607] platform-linux: event-notification: RTM_NEWROUTE, flags multi, seq 4, in-dump: type local table 255 127.0.0.0/8 via 0.0.0.0 dev 1 metric 0 mss 0 rt-src rt-kernel scope host pref-src 127.0.0.1
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2607] platform: (lo) signal: route   4   added: type local table 255 127.0.0.0/8 via 0.0.0.0 dev 1 metric 0 mss 0 rt-src rt-kernel scope host pref-src 127.0.0.1
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <trace> [1613475625.2607] platform-linux: event-notification: RTM_NEWROUTE, flags multi, seq 4, in-dump: type local table 255 127.0.0.1/32 via 0.0.0.0 dev 1 metric 0 mss 0 rt-src rt-kernel scope host pref-src 127.0.0.1
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2608] platform: (lo) signal: route   4   added: type local table 255 127.0.0.1/32 via 0.0.0.0 dev 1 metric 0 mss 0 rt-src rt-kernel scope host pref-src 127.0.0.1
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <trace> [1613475625.2608] platform-linux: event-notification: RTM_NEWROUTE, flags multi, seq 4: ignore
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2608] platform: kernel-support for RTA_PREF (ability to set router preference for IPv6 routes) detected: supported
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <trace> [1613475625.2608] platform-linux: event-notification: RTM_NEWROUTE, flags multi, seq 5, in-dump: type unicast fd42:53b4:a820:18f7::/64 via :: dev 43 metric 256 mss 0 rt-src rt-kernel
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2608] platform: (eth0) signal: route   6   added: type unicast fd42:53b4:a820:18f7::/64 via :: dev 43 metric 256 mss 0 rt-src rt-kernel
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <trace> [1613475625.2609] platform-linux: event-notification: RTM_NEWROUTE, flags multi, seq 5, in-dump: type unicast fe80::/64 via :: dev 43 metric 256 mss 0 rt-src rt-kernel
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2609] platform: (eth0) signal: route   6   added: type unicast fe80::/64 via :: dev 43 metric 256 mss 0 rt-src rt-kernel
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <trace> [1613475625.2609] platform-linux: event-notification: RTM_NEWROUTE, flags multi, seq 5, in-dump: type unicast ::/0 via fe80::216:3eff:fe77:c220 dev 43 metric 1024 mss 0 rt-src rt-ra
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2609] platform: (eth0) signal: route   6   added: type unicast ::/0 via fe80::216:3eff:fe77:c220 dev 43 metric 1024 mss 0 rt-src rt-ra
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <trace> [1613475625.2609] platform-linux: event-notification: RTM_NEWROUTE, flags multi, seq 5, in-dump: type local table 255 ::1/128 via :: dev 1 metric 0 mss 0 rt-src rt-kernel
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2610] platform: (lo) signal: route   6   added: type local table 255 ::1/128 via :: dev 1 metric 0 mss 0 rt-src rt-kernel
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <trace> [1613475625.2611] platform-linux: event-notification: RTM_NEWROUTE, flags multi, seq 5, in-dump: type local table 255 fe80::216:3eff:fec2:cefa/128 via :: dev 43 metric 0 mss 0 rt-src rt-kernel
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2611] platform: (eth0) signal: route   6   added: type local table 255 fe80::216:3eff:fec2:cefa/128 via :: dev 43 metric 0 mss 0 rt-src rt-kernel
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <trace> [1613475625.2612] platform-linux: event-notification: RTM_NEWROUTE, flags multi, seq 5, in-dump: type unicast table 255 ff00::/8 via :: dev 43 metric 256 mss 0 rt-src rt-boot
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2612] platform: (eth0) signal: route   6   added: type unicast table 255 ff00::/8 via :: dev 43 metric 256 mss 0 rt-src rt-boot
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2613] platform: kernel-support for FRA_PROTOCOL (FRA_PROTOCOL attribute for policy routing rules) detected: supported
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <trace> [1613475625.2613] platform-linux: event-notification: RTM_NEWRULE, flags multi, seq 6, in-dump: [4] 0: from all lookup 255 protocol 2
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2614] platform: kernel-support for FRA_UID_RANGE (FRA_UID_RANGE attribute for policy routing rules) not detected: assume supported
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2615] platform: kernel-support for FRA_L3MDEV (FRA_L3MDEV attribute for policy routing rules) not detected: assume supported
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2615] platform: kernel-support for FRA_IP_PROTO (FRA_IP_PROTO, FRA_SPORT_RANGE, FRA_DPORT_RANGE attributes for policy routing rules) not detected: assume supported
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2616] platform: signal: rt-rule   added: [4] 0: from all lookup 255 protocol 2
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <trace> [1613475625.2616] platform-linux: event-notification: RTM_NEWRULE, flags multi, seq 6, in-dump: [4] 32766: from all lookup 254 protocol 2
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2617] platform: signal: rt-rule   added: [4] 32766: from all lookup 254 protocol 2
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <trace> [1613475625.2617] platform-linux: event-notification: RTM_NEWRULE, flags multi, seq 6, in-dump: [4] 32767: from all lookup 253 protocol 2
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2618] platform: signal: rt-rule   added: [4] 32767: from all lookup 253 protocol 2
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <trace> [1613475625.2619] platform-linux: event-notification: RTM_NEWRULE, flags multi, seq 7, in-dump: [6] 0: from all lookup 255 protocol 2
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2619] platform: signal: rt-rule   added: [6] 0: from all lookup 255 protocol 2
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <trace> [1613475625.2620] platform-linux: event-notification: RTM_NEWRULE, flags multi, seq 7, in-dump: [6] 32766: from all lookup 254 protocol 2
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2621] platform: signal: rt-rule   added: [6] 32766: from all lookup 254 protocol 2
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <trace> [1613475625.2622] platform-linux: event-notification: RTM_NEWQDISC, flags multi, seq 8, in-dump: noqueue dev 1 family 0 handle 0 parent ffffffff info 2
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2622] platform: (lo) signal: qdisc   added: noqueue dev 1 family 0 handle 0 parent ffffffff info 2
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <trace> [1613475625.2622] platform-linux: event-notification: RTM_NEWQDISC, flags multi, seq 8, in-dump: noqueue dev 43 family 0 handle 0 parent ffffffff info 2
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2623] platform: (eth0) signal: qdisc   added: noqueue dev 43 family 0 handle 0 parent ffffffff info 2
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2628] setup NMPlatform singleton (85fe5c7e1958c870)
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2629] rules-manager: routing-rule: track [8f721f55281df2e5,0] "[4] 0: from all lookup 255 protocol 2")
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2629] rules-manager: routing-rule: track [8f721f55281df2e5,0] "[4] 32766: from all lookup 254 protocol 2")
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2629] rules-manager: routing-rule: track [8f721f55281df2e5,0] "[4] 32767: from all lookup 253 protocol 2")
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2629] rules-manager: routing-rule: track [8f721f55281df2e5,0] "[6] 0: from all lookup 255 protocol 2")
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2629] rules-manager: routing-rule: track [8f721f55281df2e5,0] "[6] 32766: from all lookup 254 protocol 2")
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2629] rules-manager: routing-rule: track [dbb17ebcd9c09fe1,0] "[4] 0: from all lookup 255 protocol 2")
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2629] rules-manager: routing-rule: track [dbb17ebcd9c09fe1,0] "[4] 32766: from all lookup 254 protocol 2")
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2629] rules-manager: routing-rule: track [dbb17ebcd9c09fe1,0] "[4] 32767: from all lookup 253 protocol 2")
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2630] rules-manager: routing-rule: track [dbb17ebcd9c09fe1,0] "[6] 0: from all lookup 255 protocol 2")
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2630] rules-manager: routing-rule: track [dbb17ebcd9c09fe1,0] "[6] 32766: from all lookup 254 protocol 2")
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2630] create NMNetns singleton (11d4d8f65b9b3fea)
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2631] auth[0x56471ce3cde0]: create auth-manager: polkit enabled
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2631] auth[0x56471ce3cde0]: set instance
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2631] setup NMAuthManager singleton (7b7e5ff4827e5232)
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <info>  [1613475625.2636] manager[0x56471ce37020]: monitoring kernel firmware directory '/lib/firmware'.
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <info>  [1613475625.2637] monitoring ifupdown state file '/run/network/ifstate'.
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2637] session-monitor: using systemd-logind session tracking
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2637] create NMSessionMonitor singleton (d3d26c0b4b469b2e)
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <trace> [1613475625.2637] dbus-object[60b705d52a70e944]: export: "/org/freedesktop/NetworkManager/AgentManager"
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <debug> [1613475625.2638] create NMAgentManager singleton (60b705d52a70e944)
Feb 16 11:40:25 debian-bullseye-nm NetworkManager[168]: <trace> [1613475625.2638] dbus-object[7510c0c7e29c8d8b]: export: "/org/freedesktop/NetworkManager/Settings"
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <info>  [1613475650.2888] hostname: hostname: hostnamed not used as proxy creation failed with: Error calling StartServiceByName for org.freedesktop.hostname1: Timeout was reached
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <info>  [1613475650.2891] hostname: hostname changed from (none) to "debian-bullseye-nm"
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.2892] create NMHostnameManager singleton (b9b00adb4bb67aad)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <trace> [1613475650.2895] policy: get-hostname: "debian-bullseye-nm"
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <trace> [1613475650.2896] policy: hostname-original: set to "debian-bullseye-nm"
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.2901] create NMFirewallManager singleton (dfbf0d028808990b)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <trace> [1613475650.2902] dns-mgr[0x56471ce17240]: creating...
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <info>  [1613475650.2911] dns-mgr[0x56471ce17240]: init: dns=default,systemd-resolved rc-manager=symlink (auto)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <trace> [1613475650.2911] dbus-object[05db2cf509db9287]: export: "/org/freedesktop/NetworkManager/DnsManager"
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.2912] create NMDnsManager singleton (05db2cf509db9287)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.2915] policy: hostname-mode: full
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.2964] setup NMManager singleton (3e2cb65b7a33a687)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <trace> [1613475650.2965] dbus-object[3e2cb65b7a33a687]: export: "/org/freedesktop/NetworkManager"
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <trace> [1613475650.2981] dns-mgr: current configuration: @aa{sv} []
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3003] create NMConnectivity singleton (0431a5410ac4f6fe)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3025] Loaded device plugin: NM6LowpanDeviceFactory (internal)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3027] Loaded device plugin: NMBondDeviceFactory (internal)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3028] Loaded device plugin: NMBridgeDeviceFactory (internal)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3031] Loaded device plugin: NMDummyDeviceFactory (internal)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3035] Loaded device plugin: NMEthernetDeviceFactory (internal)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3036] Loaded device plugin: NMInfinibandDeviceFactory (internal)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3040] Loaded device plugin: NMIPTunnelDeviceFactory (internal)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3043] Loaded device plugin: NMMacsecDeviceFactory (internal)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3123] Loaded device plugin: NMMacvlanDeviceFactory (internal)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3159] Loaded device plugin: NMPppDeviceFactory (internal)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3161] Loaded device plugin: NMTunDeviceFactory (internal)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3161] Loaded device plugin: NMVethDeviceFactory (internal)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3162] Loaded device plugin: NMVlanDeviceFactory (internal)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3163] Loaded device plugin: NMVrfDeviceFactory (internal)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3163] Loaded device plugin: NMVxlanDeviceFactory (internal)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3164] Loaded device plugin: NMWireGuardDeviceFactory (internal)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3165] Loaded device plugin: NMWpanDeviceFactory (internal)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <info>  [1613475650.3175] Loaded device plugin: NMAtmManager (/usr/lib/x86_64-linux-gnu/NetworkManager/1.28.0/libnm-device-plugin-adsl.so)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <info>  [1613475650.3220] Loaded device plugin: NMWwanFactory (/usr/lib/x86_64-linux-gnu/NetworkManager/1.28.0/libnm-device-plugin-wwan.so)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <info>  [1613475650.3230] Loaded device plugin: NMBluezManager (/usr/lib/x86_64-linux-gnu/NetworkManager/1.28.0/libnm-device-plugin-bluetooth.so)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <info>  [1613475650.3302] Loaded device plugin: NMTeamFactory (/usr/lib/x86_64-linux-gnu/NetworkManager/1.28.0/libnm-device-plugin-team.so)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <info>  [1613475650.3308] Loaded device plugin: NMWifiFactory (/usr/lib/x86_64-linux-gnu/NetworkManager/1.28.0/libnm-device-plugin-wifi.so)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3322] create NMModemManager singleton (f4f111462a3f4015)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <info>  [1613475650.3324] manager: rfkill: Wi-Fi enabled by radio killswitch; enabled by state file
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <info>  [1613475650.3330] manager: rfkill: WWAN enabled by radio killswitch; enabled by state file
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <info>  [1613475650.3337] manager: Networking is enabled by state file
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3341] dispatcher: (1) dispatching action 'hostname'
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3348] dhcp-init: enabled DHCP client 'dhclient'
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3350] dhcp-init: enabled DHCP client 'internal'
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3351] dhcp-init: enabled DHCP client 'systemd' (undocumented internal plugin)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3352] dhcp-init: enabled DHCP client 'nettools' (undocumented internal plugin)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <info>  [1613475650.3354] dhcp-init: Using DHCP client 'internal'
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3355] create NMDhcpManager singleton (2ce5e4a7466f4e61)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <trace> [1613475650.3357] settings: [timestamps-keyfile]: loaded keyfile-db for "/var/lib/NetworkManager/timestamps"
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <trace> [1613475650.3359] settings: [seen-bssids-keyfile]: loaded keyfile-db for "/var/lib/NetworkManager/seen-bssids"
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <info>  [1613475650.3363] settings: Loaded settings plugin: ifupdown ("/usr/lib/x86_64-linux-gnu/NetworkManager/1.28.0/libnm-settings-plugin-ifupdown.so")
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <info>  [1613475650.3364] settings: Loaded settings plugin: keyfile (internal)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <info>  [1613475650.3367] ifupdown: management mode: unmanaged
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <warn>  [1613475650.3368] ifupdown: interfaces file /etc/network/interfaces doesn't exist
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3370] ifupdown: unmanaged-specs: unmanaged devices count 0
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3371] ifupdown: load: no connections due to managed=false
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <trace> [1613475650.3378] settings: startup-complete: ready, no more profiles to wait for
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <trace> [1613475650.3380] config: device-state: read #43 (/run/NetworkManager/devices/43); managed=unknown, nm-owned=0, route-metric-default=0-0
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3383] device[b04a533e6a99957a] (lo): ifindex: set ifindex 1
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3384] device[b04a533e6a99957a] (lo): unmanaged: flags set to [platform-init,by-default=0x110/0x110/unmanaged/unrealized], set-unmanaged [by-default=0x100])
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <trace> [1613475650.3386] ethtool[1]: ETHTOOL_GLINK, lo: success
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3387] device[b04a533e6a99957a] (lo): constructed (NMDeviceGeneric)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3390] device[b04a533e6a99957a] (lo): start setup of NMDeviceGeneric, kernel ifindex 1
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3393] platform-linux: error reading net:/sys/class/net/lo/phys_port_id: error reading 4096 bytes from file descriptor: Operation not supported
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3395] platform-linux: sysctl: reading 'net:/sys/class/net/lo/dev_id': '0x0'
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <trace> [1613475650.3397] ethtool[1]: ETHTOOL_GDRVINFO, lo: failed: Operation not supported
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3400] platform-linux: error reading net:/sys/class/net/lo/device/sriov_numvfs: Failed to open file "device/sriov_numvfs" with openat: No such file or directory
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <trace> [1613475650.3403] ethtool[1]: ETHTOOL_GLINK, lo: success
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3404] device[b04a533e6a99957a] (lo): hw-addr: hardware address now 00:00:00:00:00:00
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3404] device[b04a533e6a99957a] (lo): hw-addr: update initial MAC address 00:00:00:00:00:00
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <info>  [1613475650.3405] device (lo): carrier: link connected
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3405] device[b04a533e6a99957a] (lo): unmanaged: flags set to [platform-init,by-default,!external-down=0x110/0x910/unmanaged/unrealized], set-managed [external-down=0x800])
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3405] device[b04a533e6a99957a] (lo): unmanaged: flags set to [by-type,platform-init,by-default,!external-down=0x118/0x918/unmanaged/unrealized], set-unmanaged [by-type=0x8])
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3406] device[b04a533e6a99957a] (lo): unmanaged: flags set to [by-type,platform-init,by-default,!sleeping,!external-down=0x118/0x919/unmanaged/unrealized], set-managed [sleeping=0x1])
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <trace> [1613475650.3406] dbus-object[b04a533e6a99957a]: export: "/org/freedesktop/NetworkManager/Devices/1"
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <info>  [1613475650.3460] manager: (lo): new Generic device (/org/freedesktop/NetworkManager/Devices/1)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3477] device[cec43923e7338dcf] (eth0): ifindex: set ifindex 43
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <trace> [1613475650.3480] ethtool[43]: ETHTOOL_GLINK, eth0: success
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3482] device[cec43923e7338dcf] (eth0): constructed (NMDeviceVeth)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3483] device[cec43923e7338dcf] (eth0): start setup of NMDeviceVeth, kernel ifindex 43
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3485] platform-linux: error reading net:/sys/class/net/eth0/phys_port_id: error reading 4096 bytes from file descriptor: Operation not supported
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3488] platform-linux: sysctl: reading 'net:/sys/class/net/eth0/dev_id': '0x0'
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <trace> [1613475650.3489] ethtool[43]: ETHTOOL_GDRVINFO, eth0: success
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3491] platform-linux: error reading net:/sys/class/net/eth0/device/sriov_numvfs: Failed to open file "device/sriov_numvfs" with openat: No such file or directory
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <trace> [1613475650.3492] ethtool[43]: ETHTOOL_GLINK, eth0: success
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3493] device[cec43923e7338dcf] (eth0): hw-addr: hardware address now 00:16:3E:C2:CE:FA
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3493] device[cec43923e7338dcf] (eth0): hw-addr: update initial MAC address 00:16:3E:C2:CE:FA
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <info>  [1613475650.3493] device (eth0): carrier: link connected
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <trace> [1613475650.3493] ethtool[43]: ETHTOOL_GSET, eth0: success
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3494] device[cec43923e7338dcf] (eth0): speed is now 10000 Mb/s
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3494] device[cec43923e7338dcf] (eth0): unmanaged: flags set to [platform-init,!by-type=0x10/0x18/unmanaged/unrealized], set-managed [by-type=0x8])
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3494] device[cec43923e7338dcf] (eth0): unmanaged: flags set to [platform-init,!sleeping,!by-type=0x10/0x19/unmanaged/unrealized], set-managed [sleeping=0x1])
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <trace> [1613475650.3494] dbus-object[cec43923e7338dcf]: export: "/org/freedesktop/NetworkManager/Devices/2"
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <info>  [1613475650.3507] manager: (eth0): new Veth device (/org/freedesktop/NetworkManager/Devices/2)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3526] create NMVpnManager singleton (1cb7311db57eefee)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3529] manager: creating virtual devices...
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <warn>  [1613475650.3530] Error: failed to open /run/network/ifstate
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3531] setting up local loopback
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3532] platform: (lo) link: setting up
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3534] platform-linux: link: change 1: flags: set 0x1/0x1 ([up] / [up])
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3538] platform-linux: do-request-link: 1 
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <trace> [1613475650.3540] platform-linux: event-notification: RTM_NEWLINK, flags 0, seq 11: 1: lo <UP,LOWER_UP;loopback,up,running,lowerup> mtu 65536 arp 772 loopback? not-init addrgenmode eui64 addr 00:00:00:00:00:00 brd 00:00:00:00:00:00 rx:0,0 tx:0,0
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <trace> [1613475650.3541] ethtool[1]: ETHTOOL_GDRVINFO, lo: failed: Operation not supported
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3542] platform-linux: do-change-link[1]: success changing link: success
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <trace> [1613475650.3554] auth: name-owner: polkit is running (now :1.2)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <trace> [1613475650.3560] firewall: D-Bus name for firewalld has no owner (firewall stopped)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <trace> [1613475650.3561] dns-sd-resolved[e8775aa10ac5a8c1]: D-Bus name for systemd-resolved has no owner
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <trace> [1613475650.3562] bluez: D-Bus name for bluez has no owner
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3567] manager: startup complete is waiting for device 'lo' (link-init)
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <debug> [1613475650.3625] dispatcher: (1) /etc/NetworkManager/dispatcher.d/01-ifupdown succeeded
Feb 16 11:40:50 debian-bullseye-nm NetworkManager[168]: <info>  [1613475650.3626] modem-manager: ModemManager available
thom311 commented 3 years ago
[1613475625.2587] platform-linux: create (ignore netns, initial netns, use udev)

this tells you that NetworkManager expects to use udev. Later you see

[1613475650.3494] device[cec43923e7338dcf] (eth0): unmanaged: flags set to [platform-init,!sleeping,!by-type=0x10/0x19/unmanaged/unrealized], set-managed [sleeping=0x1])

The device is unmanaged due to "platform-init", which means, udev did not yet declare the device as initialized.

If it would, you'd also see a message like

platform-linux: udev-add[eth0,2]: device added

which isn't there either.

Yes, /sys is mounted as rw. Consequently systemd-udevd is running (note it has ConditionPathIsReadWrite=/sys). Likewise, NetworkManager expects udev to be functional.

However, udev is not functional. I don't know how this is supposed to work. I thought, you cannot containerize udev and containers should mount /sys as read-only. See in particular https://systemd.io/CONTAINER_INTERFACE/.

lueschem commented 3 years ago

Wow - this is very helpful information. Many thanks! I will try to dig deeper.

lueschem commented 3 years ago

Indeed, LXD developers would like to see udev running within the container while the systemd(-udevd) developers do not support this use case: https://discuss.linuxcontainers.org/t/no-ipv4-on-arch-linux-containers/6395/36 https://github.com/systemd/systemd-stable/commit/bf331d87171b7750d1c72ab0b140a240c0cf32c3#r36676435

Anyway: raw.lxc: lxc.mount.auto = proc:rw sys:ro fixes the issue but might break other things (as mentioned by the LXD developers).

Bottom line: As soon as the id mapped mounts are available I can switch the edi containers into unprivileged mode and enable NetworkManager. Therefore I will keep this issue open.

thom311 commented 3 years ago

... would like to see udev running ...

Sure, it's "running". But does it work? At least some of the functionality that NetworkManager expects from udev does not work.

lueschem commented 3 years ago

@thom311: Many thanks for your clarifications! This was very helpful!

@brauner and @stgraber: I have seen similar discussions taking place about systemd-networkd. Maybe this thread helps to clarify the situation with NetworkManager. I can confirm that NetworkManager works once I switch /sys to read only using raw.lxc: lxc.mount.auto = proc:rw sys:ro. I furthermore can confirm that /sys is read only if I switch the above container to privileged mode. What kind of breakage do I have to expect if I make /sys read only within an unprivileged container? Does this affect "nesting" use cases?

stgraber commented 3 years ago

It would likely affect nesting, it will also break a bunch of tools that use /sys/class/net to configure network devices.

thom311 commented 3 years ago

It would likely affect nesting, it will also break a bunch of tools that use /sys/class/net to configure network devices.

https://systemd.io/CONTAINER_INTERFACE/ : Thus it is OK to mount the relevant sub-directories of /sys/ writable, but make sure to leave the root of /sys/ read-only.