Wrong order with systemd-remount-fs.service in zfs-mount.service

ggzengel commented 8 years ago

Because I didn't find any commit to this, is this a special undocumented feature of @FransUrbo?

In zfs-mount.service there is

Before=systemd-remount-fs.service

On boot I get:

Apr 12 20:00:06 xenserver1 systemd[1]: Found ordering cycle on systemd-remount-fs.service/start
Apr 12 20:00:06 xenserver1 systemd[1]: Found dependency on zfs-mount.service/start
Apr 12 20:00:06 xenserver1 systemd[1]: Found dependency on zfs-import-scan.service/start
Apr 12 20:00:06 xenserver1 systemd[1]: Found dependency on cryptsetup.target/start
Apr 12 20:00:06 xenserver1 systemd[1]: Found dependency on systemd-cryptsetup@VG1\x2dswap_crypt.service/start
Apr 12 20:00:06 xenserver1 systemd[1]: Found dependency on systemd-random-seed.service/start
Apr 12 20:00:06 xenserver1 systemd[1]: Found dependency on systemd-remount-fs.service/start
Apr 12 20:00:06 xenserver1 systemd[1]: Breaking ordering cycle by deleting job zfs-mount.service/start
Apr 12 20:00:06 xenserver1 systemd[1]: Job zfs-mount.service/start deleted to break ordering cycle starting with systemd-remount-fs.service/start

It couldn't work because zfs-import-scan.service -> cryptsetup.target -> systemd-cryptsetup@VG1\x2dswap_crypt.service -> systemd-random-seed.service -> systemd-remount-fs.service which means: zfs-import-scan.service -> systemd-remount-fs.service

Now you say: systemd-remount-fs.service -> zfs-mount.service -> zfs-import-scan.service which means: systemd-remount-fs.service -> zfs-import-scan.service

That's the opposite boot order.

After removing

Before=systemd-remount-fs.service

all is working as before.

**************** zfs-mount.service **************************
# /lib/systemd/system/zfs-mount.service
[Unit]
Description=Mount ZFS filesystems
DefaultDependencies=no
Wants=zfs-import-cache.service
Wants=zfs-import-scan.service
Requires=systemd-udev-settle.service
After=systemd-udev-settle.service
After=zfs-import-cache.service
After=zfs-import-scan.service
Before=local-fs.target
Before=systemd-remount-fs.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/sbin/zfs mount -a

**************** zfs-import-scan.service **************************
# /lib/systemd/system/zfs-import-scan.service
[Unit]
Description=Import ZFS pools by device scanning
DefaultDependencies=no
Requires=systemd-udev-settle.service
After=systemd-udev-settle.service
After=cryptsetup.target
ConditionPathExists=!/etc/zfs/zpool.cache

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStartPre=/sbin/modprobe zfs
ExecStart=/sbin/zpool import -d /dev/disk/by-id -aN

**************** cryptsetup.target **************************
# /lib/systemd/system/cryptsetup.target
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Encrypted Volumes
Documentation=man:systemd.special(7)

**************** systemd-cryptsetup@VG1\x2dswap_crypt.service **************************
# /run/systemd/generator/systemd-cryptsetup@VG1\x2dswap_crypt.service
# Automatically generated by systemd-cryptsetup-generator

[Unit]
Description=Cryptography Setup for %I
Documentation=man:crypttab(5) man:systemd-cryptsetup-generator(8) man:systemd-cryptsetup@.service(8)
SourcePath=/etc/crypttab
DefaultDependencies=no
Conflicts=umount.target
BindsTo=dev-mapper-%i.device
IgnoreOnIsolate=true
After=systemd-readahead-collect.service systemd-readahead-replay.service cryptsetup-pre.target
Before=cryptsetup.target
After=systemd-random-seed.service
BindsTo=dev-mapper-VG1\x2dswap.device
After=dev-mapper-VG1\x2dswap.device
Before=umount.target

[Service]
Type=oneshot
RemainAfterExit=yes
TimeoutSec=0
ExecStart=/lib/systemd/systemd-cryptsetup attach 'VG1-swap_crypt' '/dev/mapper/VG1-swap' '/dev/urandom' 'cipher=aes-xts-plain64,size=256,swap'
ExecStop=/lib/systemd/systemd-cryptsetup detach 'VG1-swap_crypt'
ExecStartPost=/sbin/mkswap '/dev/mapper/VG1-swap_crypt'

**************** systemd-random-seed.service **************************
# /lib/systemd/system/systemd-random-seed.service
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Load/Save Random Seed
Documentation=man:systemd-random-seed.service(8) man:random(4)
DefaultDependencies=no
RequiresMountsFor=/var/lib/systemd/random-seed
Conflicts=shutdown.target
After=systemd-readahead-collect.service systemd-readahead-replay.service systemd-remount-fs.service
Before=sysinit.target shutdown.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/lib/systemd/systemd-random-seed load
ExecStop=/lib/systemd/systemd-random-seed save

**************** systemd-remount-fs.service **************************
# /lib/systemd/system/systemd-remount-fs.service
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.

[Unit]
Description=Remount Root and Kernel File Systems
Documentation=man:systemd-remount-fs.service(8)
Documentation=http://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
DefaultDependencies=no
Conflicts=shutdown.target
After=systemd-readahead-collect.service systemd-readahead-replay.service systemd-fsck-root.service
Before=local-fs-pre.target local-fs.target shutdown.target
Wants=local-fs-pre.target
ConditionPathExists=/etc/fstab

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/lib/systemd/systemd-remount-fs

hex2a commented 8 years ago

+1

dechamps commented 8 years ago

I just ran into this issue after upgrading by Debian Sid system, and I don't even have cryptsetup:

zfs-mount.service: Found ordering cycle on zfs-mount.service/start
zfs-mount.service: Found dependency on systemd-udev-settle.service/start
zfs-mount.service: Found dependency on systemd-udev-trigger.service/start
zfs-mount.service: Found dependency on systemd-hwdb-update.service/start
zfs-mount.service: Found dependency on systemd-remount-fs.service/start
zfs-mount.service: Found dependency on zfs-mount.service/start
zfs-mount.service: Breaking ordering cycle by deleting job systemd-udev-settle.service/start
systemd-udev-settle.service: Job systemd-udev-settle.service/start deleted to break ordering cycle starting with zfs-mount.service/start

This prevents ZFS filesystems from being mounted on startup, because zfs-mount has a Requires dependency on systemd-udev-settle, which cannot be fulfilled.

The workaround mentioned above (removing zfs-mount's Before dependency on systemd-remount-fs.service) fixes the issue for me as well.

behlendorf commented 8 years ago

If someone has a portable fix for this it's something which could be applied to the main ZoL tree.

hex2a commented 8 years ago

this only seems to affect pkg-zfs, since the regression is introduced by this debian specific patch: https://github.com/zfsonlinux/pkg-zfs/commit/c07d163262e2b0236e450f1ab6cc20c89efab954 https://github.com/zfsonlinux/zfs/pull/3884

ggzengel commented 8 years ago

What's the problem to revert this change?

FransUrbo commented 8 years ago

Is this still a problem?

ggzengel commented 8 years ago

Sure. With every update I have to fix it.

ianabc commented 7 years ago

I'm seeing this as well, on debian jessie (8.5). Sticking the updated service file in /etc/systemd/system overrides the package installed version, e.g.

sed 's/^\(Before=systemd-remount-fs\)/#\1/g' /lib/systemd/system/zfs-mount.service > /etc/systemd/system/zfs-mount.service
systemctl daemon-reload

rmackinnon commented 7 years ago

Sabayon 15.0 system user here, this boot order issue bit me hard on the last upgrade. Since an empty /var/{lib,cache} forces a population on boot if the boot order is wrong, results in ZFS trying to then mount on a filled directory. End result != awesome, had to boot into a live cd and clear out the directories after modifying the service file as per the @ggzengel solution. I also took @ianabc suggestion and copied an altered service file into my /etc tree to keep this from happening in the future. Now I just have to remember that I did that.

thomas-zimmerman commented 4 years ago

Still a problem: Pop!_OS 19.10 --> install zfs: libzfs2linux/eoan,now 0.8.1-1ubuntu14 amd64 [installed,automatic] libzpool2linux/eoan,now 0.8.1-1ubuntu14 amd64 [installed,automatic] parted/eoan,now 3.2-26 amd64 [installed,automatic] zfs-dkms/eoan,eoan,now 0.8.1-1ubuntu14 all [installed] zfs-doc/now 0.7.12-1ubuntu5 all [installed,local] zfs-initramfs/eoan,now 0.8.1-1ubuntu14 amd64 [installed] zfs-zed/eoan,now 0.8.1-1ubuntu14 amd64 [installed,automatic] zfsutils-linux/eoan,now 0.8.1-1ubuntu14 amd64 [installed]

Reboot and:

Nov 13 19:58:09 pop-os systemd[1]: systemd-cryptsetup@cryptswap.service: Found ordering cycle on systemd-random-seed.service/start
Nov 13 19:58:09 pop-os systemd[1]: systemd-cryptsetup@cryptswap.service: Found dependency on zfs-mount.service/start
Nov 13 19:58:09 pop-os systemd[1]: systemd-cryptsetup@cryptswap.service: Found dependency on zfs-import.target/start
Nov 13 19:58:09 pop-os systemd[1]: systemd-cryptsetup@cryptswap.service: Found dependency on zfs-import-cache.service/start
Nov 13 19:58:09 pop-os systemd[1]: systemd-cryptsetup@cryptswap.service: Found dependency on cryptsetup.target/start
Nov 13 19:58:09 pop-os systemd[1]: systemd-cryptsetup@cryptswap.service: Found dependency on systemd-cryptsetup@cryptswap.service/start
Nov 13 19:58:09 pop-os systemd[1]: systemd-cryptsetup@cryptswap.service: Job systemd-random-seed.service/start deleted to break ordering cycle starting with systemd-cryptsetup@crypyptswap.service/start

I end up with no ZFS pool (and a missing /home)

mind-code commented 3 years ago

I just ran into the same problem on a fresh Ubuntu 20.04 installation with a random key swap partition.

After boot either the pool or my swap was missing, about 50/50 each. I looked around and come upon this thread and came up with the following workaround that wont need patching after each update. Its not the most elegant, but it basically works around the problem.

Basically what I did, was remove the random key swap from the /etc/fstab and /etc/crypttab and set it up manually directly after each boot via a script running from /etc/rc.local.

Steps

remove swap from /etc/fstab and /etc/crypttab and check machine comes up without swap and with mounted pool
enable rc.local (its not there by default, you can also use a systemd service file, but I went with rc.local here)

To enable rc.local create this file with root as owner and make it executable /etc/rc.local

#!/bin/sh -e

exit 0

reboot and check it worked with: systemctl status rc-local.service (if its active you can now run stuff on boot via this script)
create a small script to mount your random key swap for example in /usr/local/sbin

make_cswap

#!/bin/bash

# set umask for keyfile
umask 0377

# generate keyfile
dd if=/dev/urandom of=/run/cswap.k bs=1k count=512 >/dev/null 2>&1

# setup cryptdevice
cryptsetup --cipher aes-xts-plain64 --key-size 512 --key-file /run/cswap.k open --type plain <SWAP_DEVICE/DISK> scrypt

# mkswap / swapon
mkswap /dev/mapper/scrypt >/dev/null 2>&1
swapon /dev/mapper/scrypt

# wipe keyfile
shred -u /run/cswap.k

obviously change the mapper names and Device according to your needs (<SWAP_DEVICE/DISK> must be your desired swapdevice partition/lv)
MAKE SURE YOU SELECT THE CORRECT PARTITION/LV HERE, THE COMMANDS WILL EXECUTE AS ROOT!
this script generates random data (probably too much, but ah well)
this key is put into a tmpfs filesystem, so its only in RAM as there is no swap yet
then we create a cryptdevice with this random data as keyfile
then we create a swap device and activate it
lastly we wipe the key
test this script before you add it to the rc.local
add this script to the rc.local (below the #!/bin/sh and above the exit 0 !!!)
reboot and check

This works great for me right now and maybe this can help somebody else to work around this bug/situation.

rmackinnon commented 3 years ago

Out of curiosity, did you test your solution across suspends? In the past I've seen where there was an issue with corruption of encrypted swaps when memory is written during the suspend.

On Wed, Jul 15, 2020 at 12:13 AM mind-code notifications@github.com wrote:

I just ran into the same problem on a fresh Ubuntu 20.04 installation with a random key swap partition.

After boot either the pool or my swap was missing, about 50/50 each. I looked around and come upon this thread and came up with the following workaround that wont need patching after each update. Its not the most elegant, but it basically works around the problem.

Basically what I did, was remove the random key swap from the /etc/fstab and /etc/crypttab and set it up manually directly after each boot via a script running from /etc/rc.local. Steps

remove swap from /etc/fstab and /etc/crypttab and check machine comes up without swap and with mounted pool

enable rc.local (its not there by default, you can also use a systemd service file, but I went with rc.local here)

To enable rc.local create this file with root as owner and make it executable /etc/rc.local

!/bin/sh -e

exit 0

reboot and check it worked with: systemctl status rc-local.service (if its active you can now run stuff on boot via this script)

create a small script to mount your random key swap for example in /usr/local/sbin

make_cswap

!/bin/bash

set umask for keyfile

umask 0377

generate keyfile

dd if=/dev/urandom of=/run/cswap.k bs=1k count=512 >/dev/null 2>&1

setup cryptdevice

cryptsetup --cipher aes-xts-plain64 --key-size 512 --key-file /run/cswap.k open --type plain <SWAP_DEVICE/DISK> scrypt

mkswap / swapon

mkswap /dev/mapper/scrypt >/dev/null 2>&1 swapon /dev/mapper/scrypt

wipe keyfile

shred -u /run/cswap.k

obviously change the mapper names and Device according to your needs (<SWAP_DEVICE/DISK> must be your desired swapdevice partition/lv)

MAKE SURE YOU SELECT THE CORRECT PARTITION/LV HERE, THE COMMANDS WILL EXECUTE AS ROOT!

this script generates random data (probably too much, but ah well)

this key is put into a tmpfs filesystem, so its only in RAM as there is no swap yet

then we create a cryptdevice with this random data as keyfile

then we create a swap device and activate it

lastly we wipe the key

test this script before you add it to the rc.local

add this script to the rc.local

reboot and check

This works great for me right now and maybe this can help somebody else to work around this bug/situation.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/zfsonlinux/pkg-zfs/issues/205#issuecomment-658590367, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA4ERPPCZS2A7DBOABDGUNDR3VJK5ANCNFSM4CAXISIA .

mind-code commented 3 years ago

@rmackinnon No I did not, since for me this is a solution for a server and therefor it never suspends.

But to me it sounds like you are describing an issue in general with encrypted swaps? This swap space is no different than any other swap space. Its just that I took the automated created/mount away and do it myself manually right on boot. So the availability of swap is delayed by seconds and I assume that you dont need swap space in the first seconds after boot so the delayed creation isn't a problem, but if there is a general problem with suspend with encrypted swaps this swap/work around is probably no different.

mattiekat commented 3 years ago

@mind-code I want to one up your script because I like it and want to make it mine 😄

/etc/systemd/system/zfs-swap.service

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/root/mount-zfs-swap.sh start
ExecStop=/root/mount-zfs-swap.sh stop

[Install]
WantedBy=default.target

/root/mount-zfs-swap.sh

#! /bin/bash
set -e

KEY_SIZE=256
KEY_FILE=/tmp/swapkey
CYPHER=aes-xts-plain64
SWAP_DEVICE=<SWAP_DEVICE/DISK>  # should be something like /dev/zvol/<pool>/<vol>
CRYPT_NAME=swap_crypt

if [ "$1" == "start" ]; then
  umask 0377
  dd if=/dev/urandom of=$KEY_FILE bs=1k count=$KEY_SIZE > /dev/null 2>&1
  cryptsetup --cipher $CYPHER --key-size $KEY_SIZE --key-file $KEY_FILE open --type plain $SWAP_DEVICE $CRYPT_NAME
  mkswap /dev/mapper/$CRYPT_NAME > /dev/null 2>&1
  swapon /dev/mapper/$CRYPT_NAME
  shred -u $KEY_FILE
elif [ "$1" == "stop" ]; then
  swapoff /dev/mapper/$CRYPT_NAME
  cryptsetup close $CRYPT_NAME
else
  echo Invalid command $1
fi

stephenbfroehlich commented 3 years ago

I found an odd little workaround ... I'm encrypting swap after encrypting all of our zfs pools with a remote key that I have to paste in anyway, so mounting on boot is not an issue. What I found is that the dependency loop seems to go away if I turn set canmount=off for the root filesystem of my zfs pool. YMMV.

zfsonlinux / pkg-zfs

Wrong order with systemd-remount-fs.service in zfs-mount.service #205

Steps

!/bin/sh -e

!/bin/bash

set umask for keyfile

generate keyfile

setup cryptdevice

mkswap / swapon

wipe keyfile