search

openzfs / zfs

OpenZFS on Linux and FreeBSD
https://openzfs.github.io/openzfs-docs
Other
10.61k stars 1.75k forks source link

OpenZFS not mounting on reboot anymore #14475

Open lastb0isct opened 1 year ago

lastb0isct commented 1 year ago
Type Version/Name
Distribution Name RHEL8
Distribution Version 8_6
Kernel Version 4.18.0-372.26.1.
Architecture . x86_64
OpenZFS Version zfs-2.1.6-1 & zfs-kmod-2.1.6-1

Describe the problem you're observing

ZFS Volumes do not mount on boot. I think that there is an issue with zfs-snapshot-bootfs.service. It does not exist on my system after upgrading to 2.1.6. My dmesg shows this:

dmesg | grep zfs
[    1.581956] systemd[1]: /usr/lib/systemd/system/zfs-snapshot-bootfs.service:11: Unknown lvalue '-ExecStart' in section 'Service'
[    1.582174] systemd[1]: zfs-snapshot-bootfs.service: Service lacks both ExecStart= and ExecStop= setting. Refusing.
[    1.591136] systemd[1]: zfs-snapshot-bootfs.service: Cannot add dependency job, ignoring: Unit zfs-snapshot-bootfs.service has a ba
d unit file setting.

I have to then run systemctl restart zfs-import-scan.service & systemctl restart zfs-mount.service to mount the pools.

Describe how to reproduce the problem

Just reboot and voila nothing is mounted.

Include any warning/errors/backtraces from the system logs

This is the /var/log/messages | grep zfs:

Feb  5 17:21:05 PLEX systemd[1]: /usr/lib/systemd/system/zfs-snapshot-bootfs.service:11: Unknown lvalue '-ExecStart' in section 'Servi
ce'
Feb  5 17:21:05 PLEX systemd[1]: zfs-snapshot-bootfs.service: Service lacks both ExecStart= and ExecStop= setting. Refusing.
Feb  5 17:21:05 PLEX systemd[1]: zfs-snapshot-bootfs.service: Cannot add dependency job, ignoring: Unit zfs-snapshot-bootfs.service ha
s a bad unit file setting.
Feb  5 17:21:05 PLEX systemd-modules-load[585]: Inserted module 'zfs'
Feb  5 17:21:06 PLEX systemd[1]: /usr/lib/systemd/system/zfs-snapshot-bootfs.service:11: Unknown lvalue '-ExecStart' in section 'Servi
ce'
Feb  5 17:21:06 PLEX systemd[1]: zfs-snapshot-bootfs.service: Service lacks both ExecStart= and ExecStop= setting. Refusing.
Feb  5 17:21:09 PLEX systemd[1201]: zfs-import-scan.service: Failed to execute command: No such file or directory
Feb  5 17:21:09 PLEX systemd[1201]: zfs-import-scan.service: Failed at step EXEC spawning /usr/local/sbin/zpool: No such file or direc
tory
Feb  5 17:21:09 PLEX systemd[1]: zfs-import-scan.service: Main process exited, code=exited, status=203/EXEC
Feb  5 17:21:09 PLEX systemd[1]: zfs-import-scan.service: Failed with result 'exit-code'.
Feb  5 17:21:09 PLEX systemd[1]: zfs-env-bootfs.service: Succeeded.
Feb  5 17:21:09 PLEX systemd[1]: /usr/lib/systemd/system/zfs-snapshot-bootfs.service:11: Unknown lvalue '-ExecStart' in section 'Servi
ce'
Feb  5 17:21:09 PLEX systemd[1]: zfs-snapshot-bootfs.service: Service lacks both ExecStart= and ExecStop= setting. Refusing.
rincebrain commented 1 year ago

Looks like fallout from https://github.com/openzfs/zfs/commit/979fd5a434ebc422c02dec5deddd485ce6127fc5 if I had to guess.

@gregory-lee-bartholomew I would guess your change was incorrect, as it was "ExecStart=-..." before 8213ddff4a1b58ee8c5aa292401dbbe23a73a1c9 and your change made it "-ExecStart=...".

gregory-lee-bartholomew commented 1 year ago

It wasn't my doing. Someone else mangled the commit.

lastb0isct commented 1 year ago

The issue is that zfs-snapshot-bootfs.service doesn't even exist! It is removed so the service itself can't start...because there is not .service file.

lastb0isct commented 1 year ago 
systemctl cat zfs-snapshot-bootfs.service
No files found for zfs-snapshot-bootfs.service.
gregory-lee-bartholomew commented 1 year ago

It won't be in the systemd libraries of the running system. Dracut injects it into the initramfs from /usr/lib/dracut/modules.d/90zfs.

gregory-lee-bartholomew commented 1 year ago

As a quick fix until you can upgrade to a newer version of zfs-dracut that is fixed, you can edit the /usr/lib/dracut/modules.d/90zfs/zfs-snapshot-bootfs.service file directly and regenerate your initramfs.

gregory-lee-bartholomew commented 1 year ago

This is what mine currently looks like.

# rpm -qf /usr/lib/dracut/modules.d/90zfs/zfs-snapshot-bootfs.service
zfs-dracut-2.1.7-1.fc36.noarch

# cat /usr/lib/dracut/modules.d/90zfs/zfs-snapshot-bootfs.service
[Unit]
Description=Snapshot bootfs just before it is mounted
Requisite=zfs-import.target
After=zfs-import.target dracut-pre-mount.service
Before=dracut-mount.service
DefaultDependencies=no
ConditionKernelCommandLine=bootfs.snapshot

[Service]
Type=oneshot
ExecStart=-/bin/sh -c '. /lib/dracut-zfs-lib.sh; decode_root_args || exit; [ "$root" = "zfs:AUTO" ] && root="$BOOTFS"; SNAPNAME="$(getarg bootfs.snapshot)"; exec /sbin/zfs snapshot "$root@${SNAPNAME:-%v}"'
RemainAfterExit=yes
rincebrain commented 1 year ago

Interesting. Mismerge, then? Surprising nobody got burned by it sooner...

I'll try to remember to open a trivial fix PR for 2.1.10 later if nobody beats me to it.

gregory-lee-bartholomew commented 1 year ago

Yeah, for the record: https://github.com/openzfs/zfs/pull/13769#issuecomment-1267396763

gregory-lee-bartholomew commented 1 year ago

I'll try to remember to open a trivial fix PR for 2.1.10 later if nobody beats me to it.

I think it is already fixed in more recent versions.

rincebrain commented 1 year ago

Huh, I was looking at 2.1-release and saw it look like that I thought...nope, you're right, it's fixed in 2.1.7.

Thanks! I hadn't seen that kerfuffle with 2.1.6.

lastb0isct commented 1 year ago

Interesting that rpm -qf /usr/lib/dracut/modules.d/90zfs/zfs-snapshot-bootfs.service outputs this:

file /usr/lib/dracut/modules.d/90zfs/zfs-snapshot-bootfs.service is not owned by any package

lastb0isct commented 1 year ago

I will try and reboot the system tomorrow when i'm back in front of it. Thank you for the guidance!

gregory-lee-bartholomew commented 1 year ago

Sounds like a corrupt rpm database. Maybe rpmdb --rebuilddb would fix it?

lastb0isct commented 1 year ago

Doesn't resolve it. Shouldn't matter though, should it?

rpmdb --rebuilddb
rpm -qf /usr/lib/dracut/modules.d/90zfs/zfs-snapshot-bootfs.service
file /usr/lib/dracut/modules.d/90zfs/zfs-snapshot-bootfs.service is not owned by any package
gregory-lee-bartholomew commented 1 year ago

If the files under /var/lib/rpm are missing or corrupt, then rpm -qf may not work. Does rpm -ql zfs-dracut show anything?

# rpm -ql zfs-dracut
/usr/lib/dracut/modules.d/02zfsexpandknowledge
/usr/lib/dracut/modules.d/02zfsexpandknowledge/module-setup.sh
/usr/lib/dracut/modules.d/90zfs
/usr/lib/dracut/modules.d/90zfs/export-zfs.sh
/usr/lib/dracut/modules.d/90zfs/import-opts-generator.sh
/usr/lib/dracut/modules.d/90zfs/module-setup.sh
/usr/lib/dracut/modules.d/90zfs/mount-zfs.sh
/usr/lib/dracut/modules.d/90zfs/parse-zfs.sh
/usr/lib/dracut/modules.d/90zfs/zfs-env-bootfs.service
/usr/lib/dracut/modules.d/90zfs/zfs-generator.sh
/usr/lib/dracut/modules.d/90zfs/zfs-lib.sh
/usr/lib/dracut/modules.d/90zfs/zfs-load-key.sh
/usr/lib/dracut/modules.d/90zfs/zfs-needshutdown.sh
/usr/lib/dracut/modules.d/90zfs/zfs-rollback-bootfs.service
/usr/lib/dracut/modules.d/90zfs/zfs-snapshot-bootfs.service
/usr/share/doc/zfs-dracut
/usr/share/doc/zfs-dracut/README.dracut.markdown
lastb0isct commented 1 year ago

Yep, but not everything it seems.

rpm -ql zfs-dracut
/usr/lib/dracut/modules.d/02zfsexpandknowledge
/usr/lib/dracut/modules.d/02zfsexpandknowledge/module-setup.sh
/usr/lib/dracut/modules.d/90zfs
/usr/lib/dracut/modules.d/90zfs/export-zfs.sh
/usr/lib/dracut/modules.d/90zfs/module-setup.sh
/usr/lib/dracut/modules.d/90zfs/mount-zfs.sh
/usr/lib/dracut/modules.d/90zfs/parse-zfs.sh
/usr/lib/dracut/modules.d/90zfs/zfs-env-bootfs.service
/usr/lib/dracut/modules.d/90zfs/zfs-generator.sh
/usr/lib/dracut/modules.d/90zfs/zfs-lib.sh
/usr/lib/dracut/modules.d/90zfs/zfs-load-key.sh
/usr/lib/dracut/modules.d/90zfs/zfs-needshutdown.sh
/usr/share/doc/zfs-dracut
/usr/share/doc/zfs-dracut/README.dracut.markdown
gregory-lee-bartholomew commented 1 year ago

Very odd. Sorry, I don't know what is going on there.

rincebrain commented 1 year ago

If Plex is modifying the packages, maybe it's a question for them?

It seems kind of academic, but that would be what I'd do - go compare their specfile with the stock one.

lastb0isct commented 1 year ago

Plex is not doing anything...that is the hostname of my server, lol

rincebrain commented 1 year ago

Oh, I see. In your original paste, it had rpm -ql zfs-dracutPLEX, which confused me. I see you've edited it.

lastb0isct commented 1 year ago

Going to open a new ticket, but trying to update zfs it seems I have both dkms & kmod on my system...will ask what the best way forward is for that in a separate ticket.

lastb0isct commented 1 year ago

Updated today -- will restart when time permits.