search

openzfs / zfs

OpenZFS on Linux and FreeBSD
https://openzfs.github.io/openzfs-docs
Other
10.64k stars 1.75k forks source link

zfs-mount-generator makes systemd boot fail on Ubuntu 23.10 #15512

Open luckylinux opened 1 year ago

luckylinux commented 1 year ago

System information

Type Version/Name
Distribution Name Ubuntu
Distribution Version 23.10
Kernel Version 6.5.0-10-generic
Architecture amd64/x86_64
OpenZFS Version 2.2.0~rc3-0ubuntu4

Describe the problem you're observing

System Boot is Abnormal. Gives the message "Give root password for maintenance or press CTRL+D to continue".

Describe how to reproduce the problem

Installed Ubuntu 23.10 on NVME using debootstrap (UEFI install).

Include any warning/errors/backtraces from the system logs

journalctl.log

I tried to re-generate the zfs cache and that seems to be set correctly in /etc/zfs/zfs-list.cache/rpool.

It is possible that the /root, /var/log, /home/ folders contained some data already. I tried to empty them and reboot, to make sure that ZFS would mount correctly.

Strangely enough zfs mount -a quickly finishes (without errors) while mount -a seems to enter an infinite loop.

The culprit issue seems to be (based on log files):

Nov 10 14:00:04 UbuntuWorkstation11 systemd[1]: Reloading requested from client PID 1743 (unit emergency.service)...
Nov 10 14:00:04 UbuntuWorkstation11 systemd[1]: Reloading...
Nov 10 14:00:04 UbuntuWorkstation11 (sd-execu[1747]: /usr/lib/systemd/system-generators/zfs-mount-generator.orig failed with exit status 1.
Nov 10 14:00:04 UbuntuWorkstation11 (sd-execu[1747]: /usr/lib/systemd/system-generators/zfs-mount-generator failed with exit status 1.
Nov 10 14:00:04 UbuntuWorkstation11 ERROR: zfs-mount-generator failed and you requested a revert:
Nov 10 14:00:04 UbuntuWorkstation11 systemd[1]: Reloading finished in 469 ms.
Nov 10 14:00:04 UbuntuWorkstation11 systemd[1]: Reloading requested from client PID 1807 (unit emergency.service)...
Nov 10 14:00:04 UbuntuWorkstation11 systemd[1]: Reloading...
Nov 10 14:00:04 UbuntuWorkstation11 (sd-execu[1811]: /usr/lib/systemd/system-generators/zfs-mount-generator failed with exit status 1.
Nov 10 14:00:04 UbuntuWorkstation11 (sd-execu[1811]: /usr/lib/systemd/system-generators/zfs-mount-generator.orig failed with exit status 1.
Nov 10 14:00:04 UbuntuWorkstation11 ERROR: zfs-mount-generator failed and you requested a revert:
Nov 10 14:00:05 UbuntuWorkstation11 systemd[1]: Reloading finished in 466 ms.
Nov 10 14:00:05 UbuntuWorkstation11 systemd[1]: Reloading requested from client PID 1733 (unit emergency.service)...
Nov 10 14:00:05 UbuntuWorkstation11 systemd[1]: Reloading...
Nov 10 14:00:05 UbuntuWorkstation11 (sd-execu[1873]: /usr/lib/systemd/system-generators/zfs-mount-generator failed with exit status 1.
Nov 10 14:00:05 UbuntuWorkstation11 (sd-execu[1873]: /usr/lib/systemd/system-generators/zfs-mount-generator.orig failed with exit status 1.
Nov 10 14:00:05 UbuntuWorkstation11 ERROR: zfs-mount-generator failed and you requested a revert:
Nov 10 14:00:05 UbuntuWorkstation11 systemd[1]: Reloading finished in 467 ms.
Nov 10 14:00:12 UbuntuWorkstation11 systemd[1]: Stopped target emergency.target - Emergency Mode.

It's difficult to know where the issue lies exactly, if the script/program doesn't return more informations on where it's actually failing.

root@UbuntuWorkstation11:/# zfs list
NAME                                        USED  AVAIL  REFER  MOUNTPOINT
rpool                                      7.30G  1.75T    96K  none
rpool/ROOT                                 7.29G  1.75T    96K  none
rpool/ROOT/ubuntu                          7.29G  1.75T  6.81G  /
rpool/ROOT/ubuntu/srv                        96K  1.75T    96K  /srv
rpool/ROOT/ubuntu/tmp                       408K  1.75T   136K  /tmp
rpool/ROOT/ubuntu/usr                       424K  1.75T    96K  /usr
rpool/ROOT/ubuntu/usr/local                 328K  1.75T   192K  /usr/local
rpool/ROOT/ubuntu/var                       262M  1.75T    96K  /var
rpool/ROOT/ubuntu/var/games                  96K  1.75T    96K  /var/games
rpool/ROOT/ubuntu/var/lib                   259M  1.75T  56.4M  /var/lib
rpool/ROOT/ubuntu/var/lib/AccountsService   224K  1.75T    96K  /var/lib/AccountsService
rpool/ROOT/ubuntu/var/lib/NetworkManager    236K  1.75T   116K  /var/lib/NetworkManager
rpool/ROOT/ubuntu/var/lib/apt               123M  1.75T   122M  /var/lib/apt
rpool/ROOT/ubuntu/var/lib/dpkg             77.9M  1.75T  65.7M  /var/lib/dpkg
rpool/ROOT/ubuntu/var/log                  1.64M  1.75T   236K  /var/log
rpool/ROOT/ubuntu/var/mail                   96K  1.75T    96K  /var/mail
rpool/ROOT/ubuntu/var/spool                 264K  1.75T   144K  /var/spool
rpool/ROOT/ubuntu/var/www                    96K  1.75T    96K  /var/www
rpool/USERDATA                              524K  1.75T    96K  none
rpool/USERDATA/root                         276K  1.75T   140K  /root
rpool/USERDATA/stefano                      152K  1.75T    96K  /home/stefano

The mounting order to /root, /home/stefano and /var/tmp is suspicious. And I have a feeling that ZFS keeps trying to automount since e.g. the journalctl log file that I created in /root/ quickly disappeared after creation.

root@UbuntuWorkstation11:/# cat /etc/fstab 
# /boot Partition
UUID=8ce455bf-bc4e-4fe4-8a53-c488af3bb89c /boot ext4 auto 0 1

# /boot/efi Data
UUID=C39A-FFBB /boot/efi vfat umask=0022,fmask=0022,dmask=0022 0 1

# Tools over NFS
nas.home:/export/tools          /tools_nfs           nfs             rw,user=tools,auto,nfsvers=3          0       0
luckylinux commented 1 year ago

Tried ZFS 2.2.0 ("final") from mantic-proposed repository, but same issue happens. It seems zfs-mount-generator causes the boot process to fail, thus systemd requests Root Password or CTRL+D to continue.

Exact version: 2.2.0-0ubuntu1~23.10

Version mismatch between kernel module and system: zfs version (I'm currently trying to rebuild all DKMS modules)

root@UbuntuWorkstation02:~# zfs version
zfs-2.2.0-0ubuntu1~23.10
zfs-kmod-2.2.0~rc3-0ubuntu4
luckylinux commented 1 year ago

Version inconsistency solved, but still does NOT work

root@UbuntuWorkstation02:/# zfs version
zfs-2.2.0-0ubuntu1~23.10
zfs-kmod-2.2.0-0ubuntu1~23.10

Logs from recovery mode (kernel command line parameter "debug" set) dmesg.log journalctl.log

luckylinux commented 1 year ago

Far from ideal, but if I boot into emergency mode, then run

zfs mount -a
systemctl restart networking
systemctl restart ssh
init 3

Then I get to a normal login prompt. Login as root & run /etc/init.d/lightdm restart and I can log in as a user in MATE.

luckylinux commented 1 year ago

The man page for zfs-mount-generator is not very clear on the example.

If I run

mkdir /tmp/zfs-mount-generator
/lib/systemd/system-generators/zfs-mount-generator /tmp/zfs-mount-generator

Nothing happens. But dmesg reports zfs-mount-generator: zero or three arguments required

Trying with zero arguments yields:

root@UbuntuWorkstation02:/# /lib/systemd/system-generators/zfs-mount-generator
Input 'none' is not an absolute file system path, escaping is likely not going to be reversible.
Input 'none' is not an absolute file system path, escaping is likely not going to be reversible
luckylinux commented 1 year ago

Ubuntu Bug report, since this might be Ubuntu-specific: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/2043253

barl0g commented 8 months ago

Guys, who faced this problem. Try add "set -x" to zfs-mount-generator and execute it. This gave me an idea. Remove poolname from /etc/zfs/zfs-list.cache/. This compeltely solved this problem for me. My problem came from this: I restore my pool from backup with other name.

You can also touch newpool in this directory and exectute zfs-mount-generator as mentioned here https://manpages.debian.org/testing/zfsutils-linux/zfs-mount-generator.8.en.html

luckylinux commented 8 months ago

In my case I "solved" the problem by installing the "vanilla" openzfs package.

Script if interested - Note that 2.2.3 came out a while ago as well

#!/bin/bash
# Install Requirements
sudo apt-get --no-install-recommends install dkms
sudo apt-get install dh-dkms
sudo apt-get install aptitude libcurl4-openssl-dev libpam0g-dev lsb-release build-essential autoconf automake libtool libblkid-dev uuid-dev libudev-dev libssl-dev zlib1g-dev libaio-dev libattr1-dev libelf-dev python3 python3-dev python3-setuptools python3-cffi libffi-dev python3-packaging git libcurl4-openssl-dev debhelper-compat dh-python po-debconf python3-all-dev python3-sphinx
sudo apt-get install build-essential autoconf automake libtool gawk fakeroot libblkid-dev uuid-dev libudev-dev libssl-dev zlib1g-dev libaio-dev libattr1-dev libelf-dev python3 python3-dev python3-setuptools python3-cffi libffi-dev python3-packaging git libcurl4-openssl-dev debhelper-compat dh-python po-debconf python3-all-dev python3-sphinx
sudo apt-get install linux-headers-generic
sudo apt-get install bc bison flex

# Define Desired Version
version="2.2.2"

cd /usr/src
mkdir -p zfs
cd zfs

# Save basedir
basedir=$(pwd)

# Use git and clone zfs-$version tag
#git clone https://github.com/openzfs/zfs.git --depth 1 --tag zfs-$version
#
# Use tar (working)
wget https://github.com/openzfs/zfs/archive/refs/tags/zfs-$version.tar.gz -O zfs-$version.tar.gz
mkdir -p zfs-$version
tar xvf zfs-$version.tar.gz -C zfs-$version --strip-components 1
#
# Use tar (currently broken archive)
#wget https://github.com/openzfs/zfs/releases/download/zfs-$version/zfs-$version.tar.gz -O zfs-$version.tar.gz
#tar xvf zfs-$version.tar.gz

# Change working direectory
cd zfs-$version

# Apply Patch in order to disable SIMD and Enable successfully ZFS Compile
#wget https://raw.githubusercontent.com/chimera-linux/cports/master/main/zfs/patches/aarch64-disable-neon.patch -O aarch64-disable-neon.patch
#patch -p1 < aarch64-disable-neon.patch

sh autogen.sh
./configure
make -s -j$(nproc)
make native-deb
make native-deb-utils native-deb-dkms

# Select Subset of Packages to prevent installation of default linux-image and linux-headers
cd $basedir
mkdir -p selected-packages
mkdir -p selected-packages/$version
cd selected-packages/$version/
mv ../../openzfs-libnvpair3_$version*.deb ./
mv ../../openzfs-libpam-zfs_$version*.deb ./
mv ../../openzfs-libuutil3_$version*.deb ./
mv ../../openzfs-libzfs4_$version*.deb ./
mv ../../openzfs-libzpool5_$version*.deb ./
mv ../../openzfs-zfs-dkms_$version*.deb ./
mv ../../openzfs-zfs-initramfs_$version*.deb ./
mv ../../openzfs-zfs-zed_$version*.deb ./
mv ../../openzfs-zfsutils_$version*.deb ./

sudo apt-get install --fix-missing ./*.deb