Odroid C1 - v9.7.1 update failed (location error)

master-kw commented 2 months ago

Hi,

this morning I got an update advise (dietpi-update) for v9.7.1

Did so and got/get this failure message:

Details:

Date | 2024-08-27 11:02:03
Program name | DietPi-Patch
Command | apt-get -y --allow-change-held-packages install linux-image-edge-meson linux-dtb-edge-meson linux-u-boot-odroidc1-edge
Exit code | 100
DietPi version | v9.7.0 (MichaIng/master)
Distro version | bookworm (ID=7)
Kernel version | Linux DietPi 6.6.36-current-meson #1 SMP Thu Jun 27 11:49:15 UTC 2024 armv7l GNU/Linux
Architecture | armhf
Hardware model | Odroid C1 (armv7l) (ID=10)
Power supply | (5V 2A)
SD card | (EG: SanDisk Ultra 64 GB)
Steps to reproduce:
1. started dietpi-update
2. Expected behaviour:
update to v9.7.1
Actual behaviour:
see bug report above
Extra details:

Additional logs:

Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package linux-u-boot-odroidc1-edge

Regards, Oliver

master-kw commented 2 months ago

Here's what happened on screen:

root@DietPi:~# dietpi-update

 DietPi-Update
─────────────────────────────────────────────────────
 Phase: Checking for available DietPi update

[  OK  ] DietPi-Update | Checking IPv4 network connectivity
[  OK  ] DietPi-Update | Checking DNS resolver
[ INFO ] DietPi-Update | Getting latest version from: https://raw.githubusercontent.com/MichaIng/DietPi/master/.update/version
[  OK  ] DietPi-Update | Got valid latest version: 9.7.1
[  OK  ] DietPi-Update | Update available:
[ INFO ] DietPi-Update | Current version : v9.6.1
[ INFO ] DietPi-Update | Latest version  : v9.7.1

 DietPi-Update
─────────────────────────────────────────────────────
 Phase: Checking for update pre-requirements

[  OK  ] DietPi-Update | DietPi-Userdata validation: /mnt/dietpi_userdata
[  OK  ] DietPi-Update | Free space check: path=/ | available=56677 MiB | required=100 MiB
[ SUB1 ] DietPi-Services > stop 
[  OK  ] DietPi-Services | stop : cron

 DietPi-Update
─────────────────────────────────────────────────────
 Phase: Applying pre-patches

[  OK  ] DietPi-Update | Downloading pre-patches
[  OK  ] DietPi-Update | Applying execute permission
[  OK  ] DietPi-Update | Successfully applied pre-patches

 DietPi-Update
─────────────────────────────────────────────────────
 Phase: Upgrading APT packages

[ INFO ] DietPi-Update | APT update, please wait...
Hit:1 https://deb.debian.org/debian bookworm InRelease
Hit:2 https://deb.debian.org/debian bookworm-updates InRelease
Hit:3 https://deb.debian.org/debian-security bookworm-security InRelease
Hit:4 https://deb.debian.org/debian bookworm-backports InRelease
Hit:5 https://dietpi.com/apt bookworm InRelease
Hit:6 https://dietpi.com/apt all InRelease
Reading package lists...
[  OK  ] DietPi-Update | APT update
[ INFO ] DietPi-Update | APT upgrade, please wait...
Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
[  OK  ] DietPi-Update | APT upgrade

 DietPi-Update
─────────────────────────────────────────────────────
 Phase: Installing new DietPi code

[  OK  ] DietPi-Update | Downloading update archive
[  OK  ] DietPi-Update | Unpacking update archive
[  OK  ] DietPi-Update | Removing unused files
[  OK  ] DietPi-Update | Hardening update archive mode
[  OK  ] DietPi-Update | Installing new DietPi scripts
[  OK  ] DietPi-Update | Installing new DietPi system files
[ SUB1 ] DietPi-Set_software > verify_dietpi.txt
[  OK  ] DietPi-Set_software | Downloading current dietpi.txt
[  OK  ] verify_dietpi.txt  | Completed
[  OK  ] DietPi-Update | sync
[  OK  ] DietPi-Update | systemctl daemon-reload

 DietPi-Update
─────────────────────────────────────────────────────
 Phase: Applying incremental patches

[ INFO ] DietPi-Update | Current version : v9.6.1
[ INFO ] DietPi-Update | Latest version  : v9.7.1
[ INFO ] DietPi-Patch | Patching to DietPi v9.7...
[ INFO ] DietPi-Patch | Migrating to edge kernel to fix USB support
[ INFO ] DietPi-Patch | APT install linux-image-edge-meson linux-dtb-edge-meson linux-u-boot-odroidc1-edge, please wait...
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package linux-u-boot-odroidc1-edge
[FAILED] DietPi-Patch | APT install linux-image-edge-meson linux-dtb-edge-meson linux-u-boot-odroidc1-edge
 - Command: apt-get -y --allow-change-held-packages install linux-image-edge-meson linux-dtb-edge-meson linux-u-boot-odroidc1-edge

MichaIng commented 2 months ago

Sorry mate, I missed to add the "edge" branch U-Boot package to our APT server. Will take only 5-10 minutes.

MichaIng commented 2 months ago

...or less, done. Please retry the update 🙂.

master-kw commented 2 months ago

After a short smile... ;-)

`#### Details:

Date | 2024-08-27 18:34:28
Program name | DietPi-Patch
Command | ln -sf /boot/dtb-6.* /boot/dtb
Exit code | 1
DietPi version | v9.7.0 (MichaIng/master)
Distro version | bookworm (ID=7)
Kernel version | Linux DietPi 6.6.36-current-meson #1 SMP Thu Jun 27 11:49:15 UTC 2024 armv7l GNU/Linux
Architecture | armhf
Hardware model | Odroid C1 (armv7l) (ID=10)
Power supply | (EG: RAVPower 5V 1A)
SD card | (EG: SanDisk Ultra 16 GB)
Steps to reproduce:
1. ...
2. ...
  Expected behaviour:
...
Actual behaviour:
...
Extra details:

...

Additional logs:

ln: failed to create symbolic link '/boot/dtb/dtb-6.*': Operation not permitted

`

master-kw commented 2 months ago

PS: in the meantime I did a clean install from the scratch (for 'fun'), same result

MichaIng commented 2 months ago

Hmm, can you show the output of this:

realpath '/boot/dtb'
ls -l /boot/dtb*

master-kw commented 2 months ago

/boot/dtb
insgesamt 154
-rwxr-xr-x 1 root root  7278  2. Jul 23:36 meson6-atv1200.dtb
-rwxr-xr-x 1 root root 24685  2. Jul 23:36 meson8b-ec100.dtb
-rwxr-xr-x 1 root root 21159  2. Jul 23:36 meson8b-mxq.dtb
-rwxr-xr-x 1 root root 25081  2. Jul 23:36 meson8b-odroidc1.dtb
-rwxr-xr-x 1 root root 24789  2. Jul 23:36 meson8b-onecloud.dtb
-rwxr-xr-x 1 root root 23605  2. Jul 23:36 meson8m2-mxiii-plus.dtb
-rwxr-xr-x 1 root root 20616  2. Jul 23:36 meson8-minix-neo-x8.dtb

MichaIng commented 2 months ago

Ah, please that one, to see the actual name and in case symlink of the dtb dir(s):

ls -dl /boot/dtb*

master-kw commented 2 months ago

drwxr-xr-x 2 root root 2048 27. Aug 18:41 /boot/dtb

MichaIng commented 2 months ago

What the ... the linux-dtb-edge-meson package contains /boot/dtb-6.9.7-edge-meson. Ah wait, now I understand, on Odroid C1 /boot is a FAT partition, hence the directory is moved, not symlinked.

That directory actually seems to be from the older kernel. We definitely need to fix the postinst/postrm scripts on our Armbian fork, which do not handle kernel package uninstalls well, after having multiple kernel packages installed. I'll fix this ASAP via update patches for now.

EDIT: Ah nope, the dtb dir is what it should be.

EDIT2: This is what the preinst script does:

rm -rf /boot/dtb
rm -rf /boot/dtb-6.9.7-edge-meson

So it removes any previously present /boot/dtb, then the postinst moves the new one in place. So that should always work fine. And the new postrm does not remove /boot/dtb, the old one in your case obviously does not either. Means: we just need to remove handling possibly missing /boot/dtb for the Odroid C1.

MichaIng commented 2 months ago

Here the fix: https://github.com/MichaIng/DietPi/commit/371e958 Hence you can try again.

master-kw commented 2 months ago

Yes, this one works fine, I've made it!

Thanks a lot!

MichaIng commented 2 months ago

Thanks for testing, and sorry for not testing that patch well our end. Was a late decision to switch to edge branch, since USB ports do not work on "current" branch, and we were not able to fix this reliably. But we do not own that SBC our end.

master-kw commented 2 months ago

Now I get a continuous loop "reboot"

A reboot is required to finalise a recent kernel upgrade

MichaIng commented 2 months ago

You mean you keep seeing this also after reboot? Then the kernel image seems to not match the kernel modules. While the dtb directory was updated/moved correctly, maybe the kernel image was not 🙈. Can you check:

uname -r
ls -l /boot /lib/modules

master-kw commented 2 months ago

Yes, after each reboot.

6.9.7-edge-meson /boot: insgesamt 22994 -rwxr-xr-x 1 root root 3182 19. Feb 2024 boot.ini drwxr-xr-x 4 root root 4096 27. Aug 19:13 dietpi -rwxr-xr-x 1 root root 18092 19. Feb 2024 dietpi-LICENSE.txt -rwxr-xr-x 1 root root 16059 19. Feb 2024 dietpi-README.md -rwxr-xr-x 1 root root 17479 11. Jul 18:09 dietpi.txt -rwxr-xr-x 1 root root 3950 19. Feb 2024 dietpi-wifi.txt drwxr-xr-x 2 root root 2048 27. Aug 19:18 dtb -rwxr-xr-x 1 root root 6387358 8. Jul 09:06 initrd.img-6.6.36-current-meson -rwxr-xr-x 1 root root 10904128 2. Jul 23:36 uImage -rwxr-xr-x 1 root root 6183900 27. Aug 19:19 uInitrd /lib/modules: insgesamt 4 drwxr-xr-x 3 root root 4096 13. Jul 21:32 6.6.36-current-meson

MichaIng commented 2 months ago

My goodness, this is a mess: The kernel "uImage" seems to be edge kernel, but its kernel modules and certain other files are missing, while the "current" kernel files were remaining, which should have been removed along with the linux-image-current-meson package removal.

Can you check this:

dpkg -l | grep linux
dpkg -S /lib/modules/6.* /usr/lib/modules/6.* /boot/config-6.9.7-edge-meson

master-kw commented 2 months ago

ii  console-setup-linux         1.221                          all          Linux specific part of console-setup
ii  libselinux1:armhf           3.4-1+b6                       armhf        SELinux runtime shared libraries
ii  linux-base                  4.9                            all          Linux image base package
ii  linux-dtb-current-meson     24.8.0-trunk-dietpi2           armhf        Armbian Linux current DTBs in /boot/dtb-6.6.36-current-meson
ii  linux-image-current-meson   24.8.0-trunk-dietpi2           armhf        Armbian Linux current kernel image 6.6.36-current-meson
ii  linux-u-boot-odroidc1-edge  24.11.0-trunk-dietpi1          armhf        Das U-Boot for odroidc1
ii  util-linux                  2.38.1-5+deb12u1               armhf        miscellaneous system utilities
ii  util-linux-extra            2.38.1-5+deb12u1               armhf        interactive login tools
linux-image-current-meson: /lib/modules/6.6.36-current-meson
dpkg-query: Kein Pfad gefunden, der auf Muster /usr/lib/modules/6.6.36-current-meson passt
dpkg-query: Kein Pfad gefunden, der auf Muster /boot/config-6.9.7-edge-meson passt

MichaIng commented 2 months ago

Found it. Embarrassing mistake our end. Please repeat the update one more time with this command:

dietpi-update -1

This basically re-applies the last update, even that you are on v9.7.1 already.

master-kw commented 2 months ago

This time it really looks fine and it's up to 9.7.1. :-)

MichaIng commented 2 months ago

Good, just to be sure:

dpkg -l | grep linux
ls -l /lib/modules /boot

master-kw commented 2 months ago

ls -l /lib/modules /boot
ii  console-setup-linux         1.221                          all          Linux specific part of console-setup
ii  libselinux1:armhf           3.4-1+b6                       armhf        SELinux runtime shared libraries
ii  linux-base                  4.9                            all          Linux image base package
ii  linux-dtb-edge-meson        24.8.0-trunk-dietpi1           armhf        Armbian Linux edge DTBs in /boot/dtb-6.9.7-edge-meson
ii  linux-image-edge-meson      24.8.0-trunk-dietpi1           armhf        Armbian Linux edge kernel image 6.9.7-edge-meson
ii  linux-u-boot-odroidc1-edge  24.11.0-trunk-dietpi1          armhf        Das U-Boot for odroidc1
ii  util-linux                  2.38.1-5+deb12u1               armhf        miscellaneous system utilities
ii  util-linux-extra            2.38.1-5+deb12u1               armhf        interactive login tools
/boot:
insgesamt 27388
-rwxr-xr-x 1 root root     3182 19. Feb 2024  boot.ini
-rwxr-xr-x 1 root root   205976  2. Jul 23:36 config-6.9.7-edge-meson
drwxr-xr-x 4 root root     4096 27. Aug 19:49 dietpi
-rwxr-xr-x 1 root root    18092 19. Feb 2024  dietpi-LICENSE.txt
-rwxr-xr-x 1 root root    16059 19. Feb 2024  dietpi-README.md
-rwxr-xr-x 1 root root    17479 11. Jul 18:09 dietpi.txt
-rwxr-xr-x 1 root root     3950 19. Feb 2024  dietpi-wifi.txt
drwxr-xr-x 2 root root     2048 27. Aug 19:59 dtb
-rwxr-xr-x 1 root root  6183871 27. Aug 20:00 initrd.img-6.9.7-edge-meson
-rwxr-xr-x 1 root root  4493573  2. Jul 23:36 System.map-6.9.7-edge-meson
-rwxr-xr-x 1 root root 10904128  2. Jul 23:36 uImage
-rwxr-xr-x 1 root root  6183935 27. Aug 20:00 uInitrd

/lib/modules:
insgesamt 4
drwxr-xr-x 3 root root 4096 27. Aug 20:00 6.9.7-edge-meson

MichaIng commented 2 months ago

Great, now it is as it should be. There is definitely a beta phase needed when we do another kernel migration for Odroid C1, so I can ping some of you known owners of this SBC to test the update, before we release it. Not only do we not have this SBC, but it is also the only one with Armbian-based kernel which still has a /boot FAT partition, so a special case in this regards as well.

r3n3647 commented 2 months ago

Hello there! I updated my Odrid C1+ to DietPi v9.7.1 today and now network is not working anymore so I cannot ssh into device anymore. Till that everything worked flawless!

Addenum:

seems like no ip-adresses are assigned anymore after boot
after restarting network from dietpi-config WLAN works again OR # systemctl restart networking
ifup@wlan0.service takes over 1min to start
connection to accesspoint is established (checked with ip dev wlan0 link) but no ip request is send to dhcp-server
device runs unusual hot (>60°C /140 °F) in IDLE

Addenum II:

issue persists after completely fresh install of latest image: WLAN will not work after boot - always have to restart networking with dietpi-config to get ip adress from router OR # systemctl restart networking

MichaIng commented 2 months ago

Which WiFi adapter do you use? I am asking/wondering, since with the current kernel, the USB ports did not work at all, and we only perform the kernel upgrade, if current kernel is installed, an neither edge or legacy concurrently.

Can you show the DietPi update log, just to see whether the kernel update has been applied, and from which previous version on:

cat /var/tmp/dietpi/logs/dietpi-update.log

And WiFi works after you restart the interface?

systemctl restart ifup@wlan0

And can you show the full log for the interface setup attempt after boot?

journalctl -u ifup@wlan0

Maybe it is some timing issue. There are cases where the adapter for whatever reason takes longer to be usable at boot, while the interface is registered already, hence ifup@wlan0.service starts and tries to connect, but it cannot work yet. Usually, one would expect the driver to not make the interface available, until the adapter has been fully initialised and is usable, to avoid the need for arbitrary workarounds, but the Odroid C1 is vastly unsupported in mainline kernel and in general, so maybe a recent change in the USB or WiFi driver caused such quirk.

master-kw commented 2 months ago

@r3n3647:

I am a complete DietPi layman and was glad that I managed to get it all right.

I only have a 'tiny installation' (only AdGuardHome/Unbound) with a wired network connection and the past incident was the first thing I'd to worry about

Edit: this runs on a C1+, too. Edit2: Sooner or later I will switch to a Raspberry

r3n3647 commented 2 months ago

And WiFi works after you restart the interface?

Yes

And can you show the full log for the interface setup attempt after boot?

journalctl -u ifup@wlan0

Can you show the DietPi update log, just to see whether the kernel update has been applied, and from which previous version on.

Where is the dietpi-update.log file?

# uname -a
Linux printserver 6.9.7-edge-meson #1 SMP Thu Jun 27 11:52:32 UTC 2024 armv7l GNU/Linux

I also have minimal dietpi installation with CUPS only as a printserver.

I did another fresh install on eMMC for odroid c1 and encountered the same issue again - no WLAN after boot. Perhaps a race condition.

Meanwhile I had installed and configured a RPi0_I as replacement for my printserver. Works flawless.

master-kw commented 2 months ago

@MichaIng

Is the Odroid with this kernel update still promising for a while or should I start looking for a Raspberry?

Regards, Oliver

MichaIng commented 2 months ago

Where is the dietpi-update.log file?

cat /var/tmp/dietpi/logs/dietpi-update.log

wpa_supplicant[437]: nl80211: kernel reports: key addition failed

Hmm, this error throws some search engine results, but for various different circumstances. Does it help to delay network bring-up a bit?

mkdir /etc/systemd/system/ifup@wlan0.service.d
echo -e '[Service]\nExecStartPre=/bin/sleep 20' > /etc/systemd/system/ifup@wlan0.service.d/defer.conf

Is the Odroid with this kernel update still promising for a while or should I start looking for a Raspberry?

We were actually about to drop support for Odroid C1 already 5 years ago, when the legacy kernel form Hardkernel did not work well with newly Debian Stretch release, and mainline/Armbian kernel had no GPU/HDMI support. We however kept it as headless system then. Armbian however dropped support soon after, and Hardkernel as well. Some individual volunteers kept fixing things here and there, even HDMI again, but as can be seen from broken USB support with Linux 6.6, it is fragile at best. It runs as long it runs, but any kernel update can break things. I will form now on let every new kernel build be tested by some Odroid C1 users, before distributing it. But if there are more serious regressions, it would mean to be stuck with a non-updated kernel version, which is at some point a security issue as well.

TL;DR: If you have serious tasks, better watch out for a newer SBC.

RPi is always a safe choice, of course, but there are better choices depending on tasks/requirements. Allwinner SoC SBCs are weaker and cheaper, but usually have very quickly and good mainline Linux support, hence long support is moreless guaranteed. Rockchip SoC SBCs can be very powerful and efficient (RK3588), but Rockchip provides an own kernel branch, instead of contributing to mainline kernel, hence it takes longer before proper mainline kernel support is there, with GPU acceleration etc. But they are then usually well maintained as well, since popular.

master-kw commented 2 months ago

I only need a headless system for AdGuardHome/unbound, eventually looking forward to five or six docker container (e.g. Vaultwarden, Baikal).

MichaIng commented 2 months ago

I guess the data/configs for those Docker containers can be relatively easily transferred to a new system, if required. I suggest, to keep using the Odroid C1 as long as it works, and is powerful enough to handle the load (no problem with AdGuard Home + Unbound + vaultwarden, you might even consider our slim install options for those 3 software titles). We are doing kernel upgrades and migrations for the Odroid C1 more careful from now on, letting test everything, before shipping it. And you can keep a dietpi-backup (or other way of system or SD card backup), just in case some particular hardware has issues after the kernel upgrade, like the WiFi adapter of @r3n3647.

master-kw commented 2 months ago

Thanks for your reply. A daily/nighlty NFS-backup is already implemented, the Docker containers are still running on a Synology NAS (as long as Synology is supporting this --> aka Container Manager --> Docker Beta 24.02 --> EOL).

r3n3647 commented 2 months ago

Which WiFi adapter do you use?

0bda:8176 Realtek Semiconductor Corp. RTL8188CUS 802.11n WLAN Adapter Original 'ODROID' branded ;-)

cat /var/tmp/dietpi/logs/dietpi-update.log

File does not exist. (fresh install)

Adding 20s delay to ifup@wlan0.service does not help - connection to WLAN accesspoint is established but still no ip assigned. ifup@wlan0.service already stalls boot for over a minute.

I found a dirty workaround by disabling networking.service, masking ifup@wlan0.service (as something always triggers it also if disabled!) and install / enable dhcpcd with wpa_supplicant hook enabled. It takes quite long till network comes up - but it works for me atm.

The most simple way to 'solve' this issue would be to just downgrade to previous DietPi version 9.6 which was working fine. Unfortunately DietPi upgrades itself to latest version 9.7 after installing as soon as it can reach the internet.

Is the Odroid with this kernel update still promising for a while or should I start looking for a Raspberry?

As long as Odroid C1 is supported by any distribution I will try to use it as long as possible. Otherwise it would end up as another paperweight like my Odroid U2/U3 devices. The RaspberryPi ZeroW1 I'am using now as a printserver is very very slow and unstable.

Raspberrys are of course much better maintained than Odroids - but to buy another device as long as I have one which could do the job makes no sense to me.

master-kw commented 2 months ago

@MichaIng: FYI (with regard to test purposes), current I've swapped the Odroid C1+ for a Raspberry Pi 4 (4GB).

PS: with the RasPi 4 Adguard and Unbound are actually quite a bit faster! PSS: I will either sell it (Odroid) or keep it as an “emergency client”

MichaIng commented 2 months ago

@r3n3647 Sorry for the late reply. In case you have time and mood to test: If /var/tmp/dietpi/logs/dietpi-update.log did not exist, if was a DietPi v9.7 image already, right? If you have v9.6 image, could you apply the DietPi update there, and then check above log file? Or maybe easier, just apply an APT upgrade, paste logs (so we see the kernel version change), and check whether WiFi fails as well after reboot:

apt update
apt upgrade

master-kw commented 2 months ago

@MichaIng: I think you mean @r3n3647 . I'm currently fine with my Raspberry 4 (merged with my RasPi3, the Odroid has retired).

Oliver

MichaIng commented 2 months ago

Oh sorry, you are right 🙂.

MichaIng / DietPi