Paraphraser
commented
4 weeks ago Intriguing. In my case, the closest ARM platform I can get is:
4GB Raspberry Pi 4 Model B Rev 1.1 running Debian GNU/Linux 12 (bookworm) as full 64-bit OSThis was built using Raspberry Pi OS plus PiBuilder.
My Raspberry Pi OS starting point is always the "with desktop" image, rather than either the "with desktop and recommended software" or "lite" image variants.
Originally, there was no reason for this being my starting point other than "it seemed like a good idea at the time". I've just stuck with it ever since. If it ain't broke...
The reason why the starting point may be important is because of something I've noticed when building Proxmox guests. I start from the Debian "netinst" ISO but the final behaviour depends on options I choose during the install of the system (ie long before PiBuilder does anything). In particular, if I turn off the "Debian desktop environment" in the OS installer then Network Manager does not get installed and neither do some other useful things like the avahi daemon.
On the Pi, my rule-of-thumb has been "if it's Bookworm or later then Network Manager is active, otherwise it's 'older-style' networking".
I'd struggle to define exactly what 'older-style' really means.
Although I haven't tried it, it's possible that starting from the Raspberry Pi OS "Lite" image (which, according to the doco, has no Desktop environment) will behave like the netinst.iso on Proxmox with the desktop environment turned off: no network manager.
You may be wondering if there's a point to all the above. There is.
Way back in the early days of IOTstack on Buster, we encountered two problems:
- Pis would hang during boot-up; and
- WiFi interfaces kept disappearing.
The hang during boot-up was solved by @gbsmith - see #253 - with the initial solution being to explicitly allow DHCP on eth0 and wlan0 (implicitly denying DHCP to all other interfaces). This was later modified to the wildcard form you will see in the recommended patch, which caters for people with USB-to-Ethernet dongles. This patch is implemented by PiBuilder.
I did note, in passing, that Proxmox guests use
ensNNnaming for virtual Ethernet ports. In theory, that means the currentallowinterfacesrule would exclude those. However, those interfaces do obtain dynamic IP addresses via DHCP and I've been assuming that Proxmox must be managing the DHCP requests.
I've just done a bit more experimenting. The baseline /etc/dhcpcd.conf seems to be non-existent on Bookworm systems. But that's in the presence of Network Manager. PiBuilder adds the patch but disabling the patch results in no material difference in ip a. Under Buster (ie no Network Manager), disabling the patch results in the Docker virtual interfaces gaining link-local IP addresses. That was the original problem - that allocation process could create a race condition and the Pi would hang during boot-up.
All up, I'd say the patch is still needed for non-Network Manager systems but is ignored in Network Manager systems. I'll fix PiBuilder so it doesn't bother patching Network Manager systems.
There's background to the second problem here.
Initially, this problem seemed to be confined to WiFi interfaces but it later became apparent that Ethernet interfaces would also occasionally go walkabout so I generalised the "fix".
The fix is brute force and pretty darn unsubtle. PiBuilder only tries to activate it if Network Manager is not running and, even then, it only applies to eth0 and wlan0. It also assumes /etc/rc.local runs at boot time and my journey through Proxmox has made me realise that isn't always true.
The reason it is only needed if Network Manager is not running is because Network Manager already tries to keep interfaces active. And does a much better job of it!
Rolling all that together, I'd say:
- Check if Network Manager is running.
-
If Network Manager isn't running then:
- see if PiBuilder has installed
/usr/bin/isc-dhcp-fix.sh - see if
/etc/rc.localpurports to launchisc-dhcp-fix.shand that theeth0interface is listed on the command. -
verify that
rc.localis actually being launched. Something like:$ ps -ax | grep isc-dhcp-fix.sh | grep -v grep - make sure the
/etc/dhcpcd.confpatch is still in place because it's likely to be doing useful things.
- see if PiBuilder has installed
Now to some cold hard facts:
- I have never encountered a freezing interface on a Proxmox guest.
- I have never encountered a freezing interface on a Raspberry Pi running Bookworm.
- In both of the above, NetworkManager is active so the
isc-dhcp-fix.shmechanism is never activated by PiBuilder. - I still encounter freezing interfaces on Raspberry Pis running Bullseye but the
isc-dhcp-fix.shtakes care of the problem, both on Ethernet and WiFi.
I wanted to compare what I was seeing on my Bookworm Raspberry Pi with the steps/output in your issue.
First, is Network Manager running?
$ sudo systemctl is-active NetworkManager.service
active
$ nmcli -t -f RUNNING general
runningYes. Network Manager is running.
What is the state of the interfaces?
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether dc:a6:32:41:60:ef brd ff:ff:ff:ff:ff:ff
inet 192.168.132.100/24 brd 192.168.132.255 scope global dynamic noprefixroute eth0
valid_lft 603610sec preferred_lft 603610sec
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether dc:a6:32:41:60:f0 brd ff:ff:ff:ff:ff:ff
inet 192.168.132.101/24 brd 192.168.132.255 scope global dynamic noprefixroute wlan0
valid_lft 607839sec preferred_lft 607839sec
4: br-6b22c68d7e90: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:13:af:bb:0d brd ff:ff:ff:ff:ff:ff
inet 172.30.0.1/22 brd 172.30.3.255 scope global br-6b22c68d7e90
valid_lft forever preferred_lft forever
5: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:ae:0a:14:32 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
9: veth82a6071@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6b22c68d7e90 state UP group default
link/ether 26:0b:5c:61:1c:4a brd ff:ff:ff:ff:ff:ff link-netnsid 1
11: veth4843c76@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6b22c68d7e90 state UP group default
link/ether 12:05:10:2f:43:49 brd ff:ff:ff:ff:ff:ff link-netnsid 2
13: vethafd40c4@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6b22c68d7e90 state UP group default
link/ether 9a:7e:b1:1d:36:0a brd ff:ff:ff:ff:ff:ff link-netnsid 3
15: veth392ca19@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-6b22c68d7e90 state UP group default
link/ether 9a:1c:92:c5:a8:e7 brd ff:ff:ff:ff:ff:ff link-netnsid 0Is isc-dhcp-fix.sh running?
$ ps -ax | grep isc-dhcp-fix.sh | grep -v grep
$No! This is the expected behaviour in a system built by PiBuilder in the presence of Network Manager.
Is DHCP (theoretically) restricted to the expected Raspberry Pi physical interfaces?
$ cat /etc/dhcpcd.conf
# patch needed for IOTstack - stops RPi freezing during boot.
# see https://github.com/SensorsIot/IOTstack/issues/219
# see https://github.com/SensorsIot/IOTstack/issues/253
allowinterfaces eth*,wlan*Yes. Again, the expected behaviour with PiBuilder. And, if you go back to the ip a you'll see none of the veth interfaces has an visible IP address. The docker0 and br-* are allocated by Docker. But, if you're running Network Manager, this is irrelevant (as explained earlier).
What's the story with my "networking service"?
$ sudo systemctl status networking.service
● networking.service - Raise network interfaces
Loaded: loaded (/lib/systemd/system/networking.service; enabled; preset: enabled)
Active: active (exited) since Thu 2024-05-23 11:25:05 AEST; 1 week 6 days ago
Docs: man:interfaces(5)
Main PID: 541 (code=exited, status=0/SUCCESS)
CPU: 251msNow, I have to admit that I have never before ever considered the "networking service" and, other than the inference one can make from its name that it might have something to do with networking, I really have no idea what it does or how it fits in. In other words, although I can see it has exited, I have no idea whether that's significant. I can assure you this Pi is working despite that exit status so, on balance, I'd be inclined to ignore it.
Do I see anything about the networking service in my log?
$ sudo journalctl -xeu networking.service
-- No entries --No. How long has this Pi been up?
$ uptime
11:48:10 up 13 days, 22 min, 1 user, load average: 0.32, 0.19, 0.11Well, that sort of matches-up. Today (June 5) minus 13 days is May 23 which is what the status output is showing, so it looks like this starts and exits at boot time.
After a process of elimination, I find that I actually have to go back 55 days to find anything about this service in my journal:
$ sudo journalctl -xu networking.service --since "55 days ago"
Apr 11 11:59:02 tri-dev systemd[1]: networking.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ An ExecStart= process belonging to unit networking.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 1.
Apr 11 11:59:03 tri-dev systemd[1]: networking.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit networking.service has entered the 'failed' state with result 'exit-code'.
Apr 11 11:59:03 tri-dev systemd[1]: Failed to start networking.service - Raise network interfaces.
░░ Subject: A start job for unit networking.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit networking.service has finished with a failure.
░░
░░ The job identifier is 99 and the job result is failed.
-- Boot 50378e6b432a44d2b3c6d2ce40eda655 --
Apr 11 12:00:04 tri-dev systemd[1]: networking.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ An ExecStart= process belonging to unit networking.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 1.
Apr 11 12:00:04 tri-dev systemd[1]: networking.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit networking.service has entered the 'failed' state with result 'exit-code'.
Apr 11 12:00:04 tri-dev systemd[1]: Failed to start networking.service - Raise network interfaces.
░░ Subject: A start job for unit networking.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit networking.service has finished with a failure.
░░
░░ The job identifier is 104 and the job result is failed.
-- Boot 3d183be1547848cc88c637958a051321 --
Apr 11 15:31:29 tri-dev systemd[1]: networking.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ An ExecStart= process belonging to unit networking.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 1.
Apr 11 15:31:30 tri-dev systemd[1]: networking.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit networking.service has entered the 'failed' state with result 'exit-code'.
Apr 11 15:31:30 tri-dev systemd[1]: Failed to start networking.service - Raise network interfaces.
░░ Subject: A start job for unit networking.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit networking.service has finished with a failure.
░░
░░ The job identifier is 96 and the job result is failed.
-- Boot 22fe94cd1d7249709a7efcd8679fbef1 --
Apr 11 15:32:48 tri-dev systemd[1]: networking.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ An ExecStart= process belonging to unit networking.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 1.
Apr 11 15:32:48 tri-dev systemd[1]: networking.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit networking.service has entered the 'failed' state with result 'exit-code'.
Apr 11 15:32:48 tri-dev systemd[1]: Failed to start networking.service - Raise network interfaces.
░░ Subject: A start job for unit networking.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit networking.service has finished with a failure.
░░
░░ The job identifier is 95 and the job result is failed.That's a lot of crud to wade through so let me reduce it to what I see as its essence:
Apr 11 11:59:02 Subject: Unit process exited
Apr 11 11:59:03 Subject: Unit failed
Apr 11 11:59:03 Subject: A start job for unit networking.service has failed
-- Boot 50378e6b432a44d2b3c6d2ce40eda655 --
Apr 11 12:00:04 Subject: Unit process exited
Apr 11 12:00:04 Subject: Unit failed
Apr 11 12:00:04 Subject: A start job for unit networking.service has failed
-- Boot 3d183be1547848cc88c637958a051321 --
Apr 11 15:31:29 Subject: Unit process exited
Apr 11 15:31:30 Subject: Unit failed
Apr 11 15:31:30 Subject: A start job for unit networking.service has failed
-- Boot 22fe94cd1d7249709a7efcd8679fbef1 --
Apr 11 15:32:48 Subject: Unit process exited
Apr 11 15:32:48 Subject: Unit failed
Apr 11 15:32:48 Subject: A start job for unit networking.service has failedAll those reboots tell me I was chasing a problem and, if memory serves, the Pi had developed an intermittent fault where either I couldn't connect to it over SSH, or I'd be connected and the connection would reset.
My first suspicion was the Pi itself but I soon realised that I could connect to it just fine via its WiFi interface, so I started to focus on the Ethernet side of things: the port, the cable, the switch port.
It turned out to be a bad Ethernet patch cable. The cable wasn't new (it had been working for years). It hadn't been crunched or twisted. It tested OK for continuity. It had just somehow gone flaky. As soon as I replaced it, the problem went away.
Maybe you have a similar problem? To be honest, I don't see how that explains your "after 24 hours" but it's still something to consider.
And, on that topic, in the last six months I've had to throw out an old gigabit switch which, after a decade of faithful service, suddenly decided to reduce its transfer rate to about 5mbps. There was no evidence of any malfunction in any of the LEDs but when I picked it up to start swapping cables between ports, it was hotter than the blazes of Hell. I dropped it with a yelp and turned it off.
Hope this helps.
calcut
I've run into an issue running this on a Hetzner cloud server. Still narrowing it down, but I'm suspecting something related to DHCP that has been configured by IoTStack (or possibly pibuilder).
Basically about 24 hours after a reboot, the server loses all network connectivity. I can't ssh in, although I can get in through Hetzner's web console and see that ping 8.8.8.8 fails.
slightly more detail is provided by
sudo journalctl -xeu networking.serviceThe server is an ARM architecture running Debian 12. Using more or less the typical stuff - NodeRED, Influx, Grafana 1.x, Portainer, and Nginx Proxy Manager. But I've seen the issue with most of those containers taken down.
The main thing I've tried to fix is commenting out this line
# allowinterfaces eth*,wlan*in/etc/dhcpcd.confbut that doesn't seem to fix it.Any tips?
Other potentially relevant info