Debian Bookworm testing and upgrade script feedback

MichaIng commented 1 year ago

This issue aims to collect feedback, testing results and info around the upcoming Debian Bookworm release.

Please read our article to get an overview: T.b.d.

Additional info can be found on our wiki: https://github.com/MichaIng/DietPi/wiki/Debian-Bookworm-testing

To run the Bookworm migration script on Bullseye systems:

sudo bash <(curl -sSf 'https://raw.githubusercontent.com/MichaIng/DietPi/master/.meta/dietpi-bookworm-upgrade')

manilx commented 1 year ago

Used the script to update 6 VM's. On 4 it worked without issue on 2 it errored out.

Strange thing is they all had a base template......

manilx commented 1 year ago

Setting up grub-pc (2.06-8) ...
Replacing config file /etc/default/grub with new version
grub-pc: Running grub-install ...
/dev/sda does not exist, so cannot grub-install to it!
You must correct your GRUB install devices before proceeding:

  DEBIAN_FRONTEND=dialog dpkg --configure grub-pc
  dpkg --configure -a
dpkg: error processing package grub-pc (--configure):
 installed grub-pc package post-installation script subprocess returned error exit status 1

Joulinar commented 1 year ago

can you share following from the 2 VM's having an issue and another example from a system working

lsblk -o name,fstype,label,size,ro,type,mountpoint,partuuid,uuid

manilx commented 1 year ago

Working:

NAME    FSTYPE LABEL  SIZE RO TYPE MOUNTPOINT PARTUUID                             UUID
xvda                    8G  0 disk
|-xvda1 ext4            7G  0 part /          42e48e80-01                          7e5234cc-eb02-4544-b50b-9b14cda36655
|-xvda2                 1K  0 part            42e48e80-02
└─xvda5 swap          975M  0 part            42e48e80-05                          b77c54c2-e515-4c40-b2a5-6297b1a2a944

Not working:

NAME    FSTYPE LABEL  SIZE RO TYPE MOUNTPOINT PARTUUID                             UUID
xvda                    8G  0 disk
├─xvda1 ext4            7G  0 part /          42e48e80-01                          7e5234cc-eb02-4544-b50b-9b14cda36655
├─xvda2                 1K  0 part            42e48e80-02
└─xvda5 swap          975M  0 part            42e48e80-05                          b77c54c2-e515-4c40-b2a5-6297b1a2a944

manilx commented 1 year ago

Sorry. This is the non working.

NAME FSTYPE LABEL SIZE RO TYPE MOUNTPOINT PARTUUID UUID xvda 8G 0 disk |-xvda1 ext4 7G 0 part / 42e48e80-01 7e5234cc-eb02-4544-b50b-9b14cda36655 |-xvda2 1K 0 part 42e48e80-02 `-xvda5 swap 975M 0 part 42e48e80-05 b77c54c2-e515-4c40-b2a5-6297b1a2a944

Working: NAME FSTYPE LABEL SIZE RO TYPE MOUNTPOINT PARTUUID UUID xvda 8G 0 disk ├─xvda1 ext4 7G 0 part / 42e48e80-01 7e5234cc-eb02-4544-b50b-9b14cda36655 ├─xvda2 1K 0 part 42e48e80-02 └─xvda5 swap 975M 0 part 42e48e80-05 b77c54c2-e515-4c40-b2a5-6297b1a2a944 There's a difference in xvda5.

manilx commented 1 year ago

Actually that swap is not needed... Guess it's a leftover off debian 11 install (installed it from debian 11 iso) and then converted via dietpi script.

MichaIng commented 1 year ago

The GRUB package needs to know where it can flash GRUB to. Since we cannot know the device name you flash the image to, we use ship them with /dev/sda as most common root device name. On first GRUB update, you however needed to change this manually, if it was wrong. Looks like you did this on 4 VMs already, but not yet on 2 of them. Run manually once:

dpkg --configure grub-pc

and select the correct device name, which seems to be /dev/xvda in your case.

Since DietPi v7.5, it is adjusted automatically on first boot: https://github.com/MichaIng/DietPi/commit/d430094 So probably your two affected VMs are the only older ones. Or if the others are older as well, probably the root device name changed after first boot, e.g. if you changed some settings on the hypervisor or moved the drive to another VM.

manilx commented 1 year ago

Tried to run, got:

# dpkg --configure grub-pc dpkg: error processing package grub-pc (--configure): package grub-pc is already installed and configured Errors were encountered while processing: grub-pc

MichaIng commented 1 year ago

Try

dpkg --configure -a

or

apt -f install

or

apt update
apt install grub-pc

instead. Not sure which state your packages are exactly on each VM.

manilx commented 1 year ago

Doesn't help.

┌──(root💀Wireguard)-[~]
└─# apt -f install
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
┌──(root💀Wireguard)-[~]
└─# apt update
apt install grub-pc
Hit:1 https://deb.debian.org/debian bullseye InRelease
Get:2 https://deb.debian.org/debian bullseye-updates InRelease [44.1 kB]
Hit:3 https://deb.debian.org/debian-security bullseye-security InRelease
Get:4 https://deb.debian.org/debian bullseye-backports InRelease [49.0 kB]
Fetched 93.0 kB in 1s (97.8 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
grub-pc is already the newest version (2.06-3~deb11u5).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
┌──(root💀Wireguard)-[~]
└─# dpkg --configure -a
┌──(root💀Wireguard)-[~]
└─# dpkg --configure grub-pc
dpkg: error processing package grub-pc (--configure):
 package grub-pc is already installed and configured
Errors were encountered while processing:
 grub-pc

MichaIng commented 1 year ago

This is on a system where the Bookworm upgrade script just failed? Seems all fine. You can force the configuration with

dpkg-reconfigure grub-pc

Joulinar commented 1 year ago

Btw apt package source seems to be bullseye still.

manilx commented 1 year ago

This a bullseye system before the upgrade script (rolled back) because the update script was failing (this is why I reported here as asked on the announcement).

Joulinar commented 1 year ago

But you have the issue with Grub already on the Bullseye system?

manilx commented 1 year ago

No issues whatsoever! Runs fine.

manilx commented 1 year ago

Tried the upgrade script again. Errored out. Entered "dpkg-reconfigure grub-pc" and got

/usr/sbin/dpkg-reconfigure: grub-pc is broken or not fully installed

manilx commented 1 year ago

Now I entered "dpkg --configure -a" and I could choose the device.

Seems OK now.

Joulinar commented 1 year ago

But as shown above, dpkg --configure grub-pc was failing on Bullseye?

manilx commented 1 year ago

Yes.

Joulinar commented 1 year ago

This would mean the issue was already there on Bullseye and just popped up due to the Bookworm upgrade. Right?

manilx commented 1 year ago

Bullseye was running fine and I had no issue booting..... So I can't tell if there was an issue. Tried the upgrade script and it failed.

MichaIng commented 1 year ago

Makes all sense then. You could have done dpkg-reconfigure grub-pc already before running the script, to fix the debconf entry in the first place.

manilx commented 1 year ago

OK.... Doesn't make to me but then. As I was not experiencing any issue booting. Anyway fixed now. Can't tell what /when started it.

Joulinar commented 1 year ago

If you find time, try to restore your backup und fix grub on Bullseye before running the update. This would be helpful and appreciated.

manilx commented 1 year ago

I would of I could!! Backup is gone and new one created.

MichaIng commented 1 year ago

As I was not experiencing any issue booting.

Of course not, because GRUB is shipped with our image. It is only the GRUB package, on upgrade or reinstall, which does not know where to flash GRUB to, if the pre-configured device /dev/sda does not exist. And that package is very rarely upgraded on a stable Debian release.

Joulinar commented 1 year ago

PHP8.2 is not working on ARMv6

root@DietPi1:~# /usr/sbin/php-fpm8.2 --nodaemonize --fpm-config /etc/php/8.2/fpm/php-fpm.conf
Illegal instruction
root@DietPi1:~#

root@DietPi1:~# journalctl -u php8.2-fpm
Mar 26 18:09:34 DietPi1 systemd[1]: Starting php8.2-fpm.service - The PHP 8.2 FastCGI Process Manager...
Mar 26 18:09:34 DietPi1 systemd[1]: php8.2-fpm.service: Main process exited, code=killed, status=4/ILL
Mar 26 18:09:34 DietPi1 systemd[1]: php8.2-fpm.service: Failed with result 'signal'.
Mar 26 18:09:34 DietPi1 systemd[1]: Failed to start php8.2-fpm.service - The PHP 8.2 FastCGI Process Manager.
root@DietPi1:~#

MichaIng commented 1 year ago

Same with CLI. Probably a known issue with earlier PHP versions on ARMv6 which was not yet patched for Bookworm/PHP 8.2: https://bugs.launchpad.net/raspbian/+bug/2012833

Joulinar commented 1 year ago

Domoticz is not working on Bookworm because it requires ibssl1.1. But it is not available on Bookworm

root@DietPiOr5:~# journalctl -u domoticz.service
Apr 05 00:46:38 DietPiOr5 systemd[1]: Started domoticz.service - Domoticz (DietPi).
Apr 05 00:46:38 DietPiOr5 domoticz[2893]: /opt/domoticz/domoticz: error while loading shared libraries: libssl.so.1.1: cannot open shared object file: No such file or directory
Apr 05 00:46:38 DietPiOr5 systemd[1]: domoticz.service: Main process exited, code=exited, status=127/n/a
Apr 05 00:46:38 DietPiOr5 systemd[1]: domoticz.service: Failed with result 'exit-code'.
root@DietPiOr5:~#

MichaIng commented 1 year ago

Reported here: https://github.com/domoticz/domoticz/issues/5233 Generally support was added here: https://github.com/domoticz/domoticz/pull/5264

The problem is:

When builds are done with libssl1.1 headers, it depends on libssl1.1.
When builds are done with libssl3 headers, it depends on libssl3.
So there seems to be no way around providing two builds to not loose support for older platforms.

Not sure whether there is another solution like an intermediate/higher level library which depends on either libssl1.1 or libssl3 depending on platform which can be used to overcome this problem.

EDIT: Updated our test matrix.

Zsombe commented 1 year ago

Hello!

When i tried upgrade to Bookworm with the following command: sudo bash -c "$(curl -sSf 'https://raw.githubusercontent.com/MichaIng/DietPi/dev/.meta/dietpi-bookworm-upgrade')" i get a following error:

Hit:1 http://raspbian.raspberrypi.org/raspbian bullseye InRelease
Hit:2 https://download.docker.com/linux/raspbian bullseye InRelease
Hit:3 https://apt.sonarr.tv/debian buster InRelease
Hit:4 https://archive.raspberrypi.org/debian buster InRelease
Hit:5 https://download.mono-project.com/repo/debian raspbianbuster InRelease
Hit:6 https://repo.jellyfin.org/debian bullseye InRelease
Ign:7 https://quantum-mirror.hu/mirrors/pub/raspbian bullseye InRelease
Err:8 https://quantum-mirror.hu/mirrors/pub/raspbian bullseye Release
  404  Not Found [IP: 91.120.148.107 443]
Reading package lists...    
E: The repository 'https://quantum-mirror.hu/mirrors/pub/raspbian bullseye Release' does not have a Release file.
[FAILED] dietpi-bookworm-upgrade | APT update
 - Command: apt-get -y -eany update

What is the problem?

Joulinar commented 1 year ago

It doesn't make sense to post your issue twice. Pls try to stay with your individual issue for now. #6322

rickcrash commented 1 year ago

Hi,

updated today via the script. Almost everthing worked fine. I just had a little hassle with my old ubound which I installed long time ago by hand.

After the final reboot I found some services masked.

systemctl list-unit-files --type=service | grep mask

alsa-utils.service                     masked          enabled
apt-daily-upgrade.service              masked          enabled
apt-daily.service                      masked          enabled
cloudprintd.service                    masked          enabled
collectd.service                       masked          enabled
containerd.service                     masked          enabled
cryptdisks-early.service               masked          enabled
cryptdisks.service                     masked          enabled
docker.service                         masked          enabled
getty-static.service                   masked          enabled
hwclock.service                        masked          enabled
mdadm-waitidle.service                 masked          enabled
mdadm.service                          masked          enabled
mosquitto.service                      masked          enabled
nfs-common.service                     masked          enabled
ntp.service                            masked          enabled
rc.service                             masked          enabled
rcS.service                            masked          enabled
screen-cleanup.service                 masked          enabled
serial-getty@ttyS0.service             masked          enabled
serial-getty@ttyS1.service             masked          enabled
serial-getty@ttyS2.service             masked          enabled
serial-getty@ttyS3.service             masked          enabled
sudo.service                           masked          enabled
ups-monitor.service                    masked          enabled
x11-common.service                     masked          enabled

What's the reason for this masking?

Can I unmask some services? I already tried to unmask mdadm, without succes.

btw: Thanks a lot!

MichaIng commented 1 year ago

Most of them are masked automatically by systemd to prevent the obsolete SysV/init.d services from starting.
The apt-daily ones are masked on DietPi by default since we have our own daily APT update check implemented.
cloudprintd is this old Google printing service which does not exist anymore. You should be able to unmask and in case remove it: systemctl unmask cloudprintd; apt autopurge cloudprint-service: #3860
collectd must have been manually installed by you: https://github.com/collectd/collectd
Not sure why you have Docker masked? If you do not currently run any containers but want to keep Docker installed, that is probably a good way to prevent it from unintentionally starting up.
Same with Mosquitto.
If you do not use NTP, better just remove it: systemctl unmask ntp; apt autopurge ntp
Not sure what ups-monitor is.
The serial-getty services are masked to prevent login prompts on those UART devices. This has no effect if there are anyway no UART adapters attached, but generally it makes sense to not allow logins and have some waiting service for this on UART devices which you never intend to use.

whyisthisbroken commented 1 year ago

everything works... thanks for this easy script. very cool. stupid question, why wasn't python 3.10 updated to 3.11? the home assistant is completely reinstalled but in 3.10.11... is this normal ?

the only thing I have is as described here -> https://github.com/MichaIng/DietPi/pull/6103#issuecomment-1507296719

Joulinar commented 1 year ago

HA is running within its own pyenv which we set to 3.10.

MichaIng commented 1 year ago

Jep, I think we'll switch it to 3.11 once Bookworm is released. For now there are still some issues and the official docs and also HA's own images/containers all use Python 3.10: https://github.com/home-assistant/core/issues?q=is%3Aissue+is%3Aopen+Python+3.11

rickcrash commented 1 year ago

Most of them are masked automatically by systemd to prevent the obsolete SysV/init.d services from starting.

The apt-daily ones are masked on DietPi by default since we have our own daily APT update check implemented.

cloudprintd is this old Google printing service which does not exist anymore. You should be able to unmask and in case remove it: systemctl unmask cloudprintd; apt autopurge cloudprint-service: DietPi-Software | CloudPrint: Remove from software options #3860 ...

Thank you for your fast reply. I followed all your advices.

One service I really need is mdadm. my raid5-device was unavailable when mdadm was masked. Is there another way to use the raid5-device without unmasking mdadm?

ups-monitor is a program for monitoring and controlling a UPS. I have unmasked this, but it does not run yet.

MichaIng commented 1 year ago

I don't know much about mdadm, I think @StephanStS can verify whether this service is required to run a RAID generated with the mdadm command.

However, if it is needed for your RAID, unmask it. It is not one of the services masked by systemd and it is neither one masked by the DietPi installer. So I don't know why it is masked on your system (as with Docker and Mosquitto).

StephanStS commented 1 year ago

On my Bullseye RAID 5 system,

no mdadm service is shown via dietpi-services
the mdadm.service is not active
in htop it is shown as a process

I installed my RAID system like described in the blog post: Using a RAID system with DietPi – part 1 and following posts.

grafik

root@raid:~# systemctl status mdadm
● mdadm.service
     Loaded: masked (Reason: Unit mdadm.service is masked.)
     Active: inactive (dead)

grafik

MichaIng commented 1 year ago

Just checked it, the mdadm package itself ships the mask, in a way that it cannot even be unmasked, and an init.d service. Hence this service is only needed when using SysV as init system (resp. not systemd) but explicitly not needed when using systemd, hence the mask is fully intended.

rickcrash commented 1 year ago

Thank you for your help, i'll check the assembly of my raid5 at this weekend.

-------- Ursprüngliche Nachricht -------- Von: MichaIng @.> Gesendet: 27. April 2023 21:28:59 MESZ An: MichaIng/DietPi @.> CC: Rick @.>, Comment @.> Betreff: Re: [MichaIng/DietPi] Debian Bookworm testing and upgrade script feedback (Issue #6227)

Just checked it, the mdadm package itself ships the mask, in a way that it cannot even be unmasked, and an init.d service. Hence this service is only needed when using SysV as init system (resp. not systemd) but explicitly not needed when using systemd, hence the mask is fully intended.

-- Reply to this email directly or view it on GitHub: https://github.com/MichaIng/DietPi/issues/6227#issuecomment-1526247269 You are receiving this because you commented.

Message ID: @.***> -- Diese Nachricht wurde von meinem GrapheneOS-Gerät mit K-9 Mail gesendet.

rickcrash commented 1 year ago

I have restarted the server with the mdadm package masked. The raid device was not assembled. If mdadm is unmasked, it is not assembled either.

Manually running mdadm --assemble --scan is no problem though, I don't start the server that often.

Would an update-initramfs -u help?

Joulinar commented 1 year ago

why not unmasking mdadm.service?

MichaIng commented 1 year ago

The RAID topic has actually nothing to do with Bookworm, does it? @rickcrash please open a new issue about this so we do not further spam it here.

Some input:

I need to setup a RAID myself to efficiently check how this mdadm process is supposed to be started. At least when following our guide, the mdadm.service is not the intended way, or more precisely: With systemd, the /etc/init.d/mdadm SysV service is not intended to do so (otherwise the package wouldnt' explicitly ship the mask), but some other mechanism, probably even systemd itself if a /dev/mdX fstab entry is present.

Would an update-initramfs -u help?

You can try it, but I cannot imagine what the initramfs should have to do with the RAID, which is assembled long after the initramfs has finished its task, as long as you did not create one yourself which explicitly aims to mount the RAID for whatever reason.

@Joulinar The package itself does not ship a systemd service but only the mentioned SysV service and explicitly a mask (symlink to /dev/null) at /lib/systemd/system/mdadm.service to mask that SysV service if systemd is used. This cannot be "unmasked" via systemctl unmask mdadm. Of course the mask could be manually removed, but this would explicitly break the intended behaviour of the package, so is probably a bad idea.

rickcrash commented 1 year ago

Hi,

before updating to bookworm the RAID was always assembled. After the update no more, so I think it has to do with the update.

I tried update-initramfs -u an it worked, the raid runs after rebooting, so for me the RAID-issue is closed.

I updated 3 dietpi

one raspberry 1 with almost no extra software and the script worked perfect
one raspberry 3 with some extra software and the script worked perfect
one x86-Server with a lot of extra software as written above.

Thank you once again.

MichaIng commented 1 year ago

Hmm, if update-initramfs -u made a difference, then the kernel module was not enabled. Interesting, usually these are enabled on demand. However, good that it works when you have the module loaded in initramfs.

tmznix commented 1 year ago

Hi!

Pi-hole is failing to install due to PHP 8.2 apparently not working on ARMv6.

Tested on a Pi Zero W.

Thank you!

MichaIng commented 1 year ago

Indeed, known issue, needs a fix at Raspbian: https://bugs.launchpad.net/raspbian/+bug/2012833

aszurnasirpal commented 1 year ago

After I moved to Bookworm via script on rpi4 I couldn't run any longer - any docker-compose

I got the following error:

dietpi@DietPi:/opt/homepage$ sudo docker-compose restart
Traceback (most recent call last):
  File "/usr/lib/python3.11/importlib/metadata/__init__.py", line 563, in from_name
    return next(cls.discover(name=name))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/docker-compose", line 33, in <module>
    sys.exit(load_entry_point('docker-compose==1.29.2', 'console_scripts', 'docker-compose')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/bin/docker-compose", line 22, in importlib_load_entry_point
    for entry_point in distribution(dist_name).entry_points
                       ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/importlib/metadata/__init__.py", line 981, in distribution
    return Distribution.from_name(distribution_name)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/importlib/metadata/__init__.py", line 565, in from_name
    raise PackageNotFoundError(name)
importlib.metadata.PackageNotFoundError: No package metadata was found for docker-compose

Reinstalling via dietpi-software doesn't help

MichaIng / DietPi

Debian Bookworm testing and upgrade script feedback #6227