fix(crypt): close crypt devices to release encryption keys get on shutdown

[DanWin]

Changes

This pull request adds a shutdown hook to close encrypted devices and wipe their encryption keys from kernel memory. Cryptsetup may not be installed on every system, if systemd-cryptsetup is used. That's why I made this feature optional in order not to break any running systems.

Checklist

• [X] I have tested it locally
• [ ] I have reviewed and updated any documentation if relevant
• [ ] I am providing new code and test(s) for it

Fixes #997 #1888

[LaszloGombos]

Error: commit_message:1:1: error: use of type tag that's neither 'feat', 'fix' nor whitelisted (build, chore, ci, docs, perf, refactor, revert, style, test, improvement)

Please fix commit message to e.g.

fix(crypt): close crypt devices, so that encryption keys get released on shutdown

[DanWin]

@LaszloGombos alright, just updated it.

[LaszloGombos]

Error: commit_message:1:81: error: commit message's subject exceeds line length of 80 by 1 characters fix(crypt): close crypt devices, so that encryption keys get released on shutdown

[DanWin]

Updated once more

[LaszloGombos]

@adrelanos I appreciate your thought/review on this. Thanks !

[LaszloGombos]

Cryptsetup may not be installed on every system, if systemd-cryptsetup is used.

Is this actually an issue also with systemd-cryptsetup ? Wouldn't adding "x-initrd.attach" to /etc/crypttab for the device holding the root fs solve the issue if systemd and systemd-cryptsetup in the initramfs ?

https://www.freedesktop.org/software/systemd/man/crypttab.html#x-initrd.attach

[DanWin]

Promising option, I've just tried adding it to my crypttab. However, it looks like it didn't change anything. At the time crypt-shutdown.sh got run, the encrypted device was still attached.

[LaszloGombos]

Promising option, I've just tried adding it to my crypttab. However, it looks like it didn't change anything. At the time crypt-shutdown.sh got run, the encrypted device was still attached.

Not sure if this is helpful, but just in case - https://github.com/systemd/systemd/issues/25538

[adrelanos]

Works nice with a Debian bookworm installed using Debian Installer, which sets up an encrypted LVM with unencrypted /boot.

Broken with Debian bookworm (actually Kicksecure) installed with Calamares, which sets up an encrypted /boot but without LVM. Debugged as much as I could. Got some udev issue. Maybe something a simple as a missing dracut module?

sudo dracut -f

dracut: Executing: /usr/bin/dracut -f
dracut: dracut module 'mksh' will not be installed, because command 'mksh' could not be found!
dracut: dracut module 'systemd-coredump' will not be installed, because command 'coredumpctl' could not be found!
dracut: dracut module 'systemd-coredump' will not be installed, because command '/usr/lib/systemd/systemd-coredump' could not be found!
dracut: dracut module 'systemd-portabled' will not be installed, because command 'portablectl' could not be found!
dracut: dracut module 'systemd-portabled' will not be installed, because command '/usr/lib/systemd/systemd-portabled' could not be found!
dracut: dracut module 'systemd-resolved' will not be installed, because command 'resolvectl' could not be found!
dracut: dracut module 'systemd-resolved' will not be installed, because command '/usr/lib/systemd/systemd-resolved' could not be found!
dracut: dracut module 'systemd-timesyncd' will not be installed, because command '/usr/lib/systemd/systemd-timesyncd' could not be found!
dracut: dracut module 'dbus-broker' will not be installed, because command 'dbus-broker' could not be found!
dracut: dracut module 'rngd' will not be installed, because command 'rngd' could not be found!
dracut: 62bluetooth: Could not find any command of '/usr/lib/bluetooth/bluetoothd /usr/libexec/bluetooth/bluetoothd'!
dracut: dracut module 'btrfs' will not be installed, because command 'btrfs' could not be found!
dracut: dracut module 'dmraid' will not be installed, because command 'dmraid' could not be found!
dracut: dracut module 'mdraid' will not be installed, because command 'mdadm' could not be found!
dracut: dracut module 'multipath' will not be installed, because command 'multipath' could not be found!
dracut: dracut module 'pcsc' will not be installed, because command 'pcscd' could not be found!
dracut: dracut module 'tpm2-tss' will not be installed, because command 'tpm2' could not be found!
dracut: dracut module 'nvmf' will not be installed, because command 'nvme' could not be found!
dracut: dracut module 'biosdevname' will not be installed, because command 'biosdevname' could not be found!
dracut: dracut module 'memstrack' will not be installed, because command 'memstrack' could not be found!
dracut: memstrack is not available
dracut: If you need to use rd.memdebug>=4, please install memstrack and procps-ng
dracut: *** Including module: systemd ***
dracut: *** Including module: systemd-initrd ***
dracut: *** Including module: modsign ***
dracut: *** Including module: console-setup ***
dracut: *** Including module: i18n ***
dracut: *** Including module: ram-wipe-exit ***
dracut: *** Including module: cold-boot-attack-defense ***
dracut: *** Including module: crypt ***
dracut: *** Including module: dm ***
dracut: Skipping udev rule: 10-dm.rules
dracut: Skipping udev rule: 13-dm-disk.rules
dracut: Skipping udev rule: 64-device-mapper.rules
dracut: *** Including module: kernel-modules ***
dracut: *** Including module: kernel-modules-extra ***
dracut: *** Including module: lvm ***
dracut: Skipping udev rule: 11-dm-lvm.rules
dracut: Skipping udev rule: 64-device-mapper.rules
dracut: *** Including module: nvdimm ***
dracut: *** Including module: overlay-root ***
dracut: *** Including module: qemu ***
dracut: *** Including module: lunmask ***
dracut: *** Including module: rootfs-block ***
dracut: *** Including module: terminfo ***
dracut: *** Including module: udev-rules ***
dracut: Skipping udev rule: 40-redhat.rules
dracut: Skipping udev rule: 91-permissions.rules
dracut: Skipping udev rule: 80-drivers-modprobe.rules
dracut: *** Including module: virtiofs ***
dracut: *** Including module: dracut-systemd ***
dracut: *** Including module: usrmount ***
dracut: *** Including module: base ***
dracut: *** Including module: fs-lib ***
dracut: *** Including module: shutdown ***
dracut: *** Including modules done ***
dracut: *** Installing kernel module dependencies ***
dracut: *** Installing kernel module dependencies done ***
dracut: *** Resolving executable dependencies ***
dracut: *** Resolving executable dependencies done ***
dracut: *** Hardlinking files ***
dracut: Mode:                     real
dracut: Method:                   sha256
dracut: Files:                    1154
dracut: Linked:                   1 files
dracut: Compared:                 0 xattrs
dracut: Compared:                 108 files
dracut: Saved:                    690 B
dracut: Duration:                 0.012228 seconds
dracut: *** Hardlinking files done ***
dracut: *** Generating early-microcode cpio image ***
dracut: *** Store current command line parameters ***
dracut: *** Stripping files ***
dracut: *** Stripping files done ***
dracut: *** Creating image file '/boot/initrd.img-6.1.0-17-amd64' ***
dracut: Using auto-determined compression method 'pigz'
dracut: *** Creating initramfs image file '/boot/initrd.img-6.1.0-17-amd64' done ***

Modified the script to gather more debug output.

[ 1163.100945] dracut Warning: Killing all remaining processes
dracut Warning: Killing all remaining processes
[ 1163.277065] dracut Warning: Unmounted /oldroot.
dracut Warning: Unmounted /oldroot.
[ 1163.287714] dracut Warning: crypt-shutdown: START
dracut Warning: crypt-shutdown: START

[ 1163.288239] dracut Warning: crypt-shutdown: systemctl status udev.service
dracut Warning: crypt-shutdown: systemctl status udev.service
Failed to connect to bus: Connection refused

[ 1163.293182] dracut Warning: crypt-shutdown: dmsetup ls --target crypt
dracut Warning: crypt-shutdown: dmsetup ls --target crypt
luks-dabd0f40-88e7-48eb-aa94-88b99ba059da   (254, 0)

[ 1163.295272] dracut Warning: crypt-shutdown: lsblkl
dracut Warning: crypt-shutdown: lsblkl
NAME                                       MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
sda                                          8:0    0   20G  0 disk
|-sda1                                       8:1    0  300M  0 part
|-sda2                                       8:2    0 11.3G  0 part
| `-luks-dabd0f40-88e7-48eb-aa94-88b99ba059da
|                                          254:0    0 11.3G  0 crypt
`-sda3                                       8:3    0  8.4G  0 part
sr0                                         11:0    1    1G  0 rom

[ 1163.302719] dracut Warning: crypt-shutdown: df -h
dracut Warning: crypt-shutdown: df -h

No output. Good.

Filesystem      Size  Used Avail[ 1163.304600] dracut Warning: crypt-shutdown: mount
 Use% Mounted on
devtmpfs        4.0M     0  4.0M   0% /dev
tmpfs           2.0G     0  2.0G   0% /oldsys/dev/shm
tmpfs           780M  146M  634M  19% /run
tmpfs           5.0M  8.0K  5.0M   1% /oldsys/run/lock

dracut Warning: crypt-shutdown: mount

proc on /oldsys/proc type proc (rw,nosuid,nodev,noexec,relatime)
sysfs on /oldsys/sys type sysfs (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /oldsys/dev type devtmpfs (rw,nosuid,size=4096k,nr_inodes=492983,mode=755,inode64)
securityfs on /oldsys/sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /oldsys/dev/shm type tmpfs (rw,nosuid,nodev,inode64)
devpts on /oldsys/dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /oldsys/run type tmpfs (rw,nosuid,nodev,size=798532k,nr_inodes=819200,mode=755,inode64)
tmpfs on /oldsys/run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /oldsys/sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /oldsys/sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /oldsys/sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
bpf on /oldsys/sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
hugetlbfs on /oldsys/dev/hugepages type hugetlbfs (rw,relatime,pagesize=2M)
mqueue on[ 1163.317630] dracut Warning: crypt-shutdown: vgdisplay
 /oldsys/dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
tracefs on /oldsys/sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
configfs on /oldsys/sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
fusectl on /oldsys/sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
tmpfs on / type tmpfs (rw,nosuid,nodev,size=798532k,nr_inodes=819200,mode=755,inode64)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,size=4096k,nr_inodes=492983,mode=755,inode64)
tmpfs on /run type tmpfs (rw,nosuid,nodev,size=798532k,nr_inodes=819200,mode=755,inode64)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)

dracut Warning: crypt-shutdown: vgdisplay
[ 1163.384809] dracut Warning: crypt-shutdown: lsof /dev/mapper/luks-dabd0f40-88e7-48eb-aa94-88b99ba059da

dracut Warning: crypt-shutdown: lsof /dev/mapper/luks-dabd0f40-88e7-48eb-aa94-88b99ba059da
[ 1163.409785] dracut Warning: crypt-shutdown: findmnt /dev/mapper/luks-dabd0f40-88e7-48eb-aa94-88b99ba059da

dracut Warning: crypt-shutdown: findmnt /dev/mapper/luks-dabd0f40-88e7-48eb-aa94-88b99ba059da

No output. Good.

[ 1163.413033] dracut Warning: crypt-shutdown: cryptsetup status luks-dabd0f40-88e7-48eb-aa94-88b99ba059da
dracut Warning: crypt-shutdown: cryptsetup status luks-dabd0f40-88e7-48eb-aa94-88b99ba059da
/dev/mapper/luks-dabd0f40-88e7-48eb-aa94-88b99ba059da is active.
  type:    LUKS1
  cipher:  aes-xts-plain64
  keysize: 512 bits
  key location: dm-crypt
  device:  /dev/sda2
  sector size:  512
  offset:  4096 sectors
  size:    23739353 sectors
  mode:    read/write

[ 1163.425169] dracut Warning: crypt-shutdown: cryptsetup --debug close luks-dabd0f40-88e7-48eb-aa94-88b99ba059da
dracut Warning: crypt-shutdown: cryptsetup --debug close luks-dabd0f40-88e7-48eb-aa94-88b99ba059da
# cryptsetup 2.6.1 processing "cryptsetup --debug close luks-dabd0f40-88e7-48eb-aa94-88b99ba059da"
# Verifying parameters for command close.
# Running command close.
# Installing SIGINT/SIGTERM handler.
# Unblocking interruption on signal.
# Allocating crypt device context by device luks-dabd0f40-88e7-48eb-aa94-88b99ba059da.
# Initialising device-mapper backend library.
# dm version   [ opencount flush ]   [16384] (*1)
# dm versions   [ opencount flush ]   [16384] (*1)
# Detected dm-ioctl version 4.47.0.
# Detected dm-crypt version 1.24.0.
# Device-mapper backend running with UDEV support enabled.
# dm status luks-dabd0f40-88e7-48eb-aa94-88b99ba059da  [ opencount noflush ]   [16384] (*1)
# Releasing device-mapper backend.
# Trying to open and read device /dev/sda2 with direct-io.
# Allocating context for crypt device /dev/sda2.
# Trying to open and read device /dev/sda2 with direct-io.
# Initialising device-mapper backend library.
# dm versions   [ opencount flush ]   [16384] (*1)
# dm table luks-dabd0f40-88e7-48eb-aa94-88b99ba059da  [ opencount flush securedata ]   [16384] (*1)
# Trying to open and read device /dev/sda2 with direct-io.
# dm versions   [ opencount flush ]   [16384] (*1)
# dm deps luks-dabd0f40-88e7-48eb-aa94-88b99ba059da  [ opencount flush ]   [16384] (*1)
# Crypto backend (OpenSSL 3.0.11 19 Sep 2023 [default][legacy]) initialized in cryptsetup library version 2.6.1.
# Detected kernel Linux 6.1.0-17-amd64 x86_64.
# PBKDF pbkdf2-sha256, time_ms 2000 (iterations 0).
# Reading LUKS header of size 1024 from device /dev/sda2
# Key length 64, device size 23743449 sectors, header size 4036 sectors.
# Deactivating volume luks-dabd0f40-88e7-48eb-aa94-88b99ba059da.
# dm versions   [ opencount flush ]   [16384] (*1)
# dm status luks-dabd0f40-88e7-48eb-aa94-88b99ba059da  [ opencount noflush ]   [16384] (*1)
# dm versions   [ opencount flush ]   [16384] (*1)
# dm table luks-dabd0f40-88e7-48eb-aa94-88b99ba059da  [ opencount flush securedata ]   [16384] (*1)
# Trying to open and read device /dev/sda2 with direct-io.
# dm versions   [ opencount flush ]   [16384] (*1)
# Udev cookie 0xd4dd417 (semid 3) created
# Udev cookie 0xd4dd417 (semid 3) incremented to 1
# Udev cookie 0xd4dd417 (semid 3) incremented to 2
# Udev cookie 0xd4dd417 (semid 3) assigned to REMOVE task(2) with flags DISABLE_LIBRARY_FALLBACK         (0x20)
# dm remove luks-dabd0f40-88e7-48eb-aa94-88b99ba059da  [ opencount flush retryremove ]   [16384] (*1)
# Udev cookie 0xd4dd417 (semid 3) decremented to 1
# Udev cookie 0xd4dd417 (semid 3) waiting for zero

/usr/lib/dracut/modules.d/90crypt/module-setup.sh

    inst_multiple -o cryptsetup lsblk lsof df findmnt vgdisplay

/usr/lib/dracut/modules.d/90crypt/crypt-shutdown.sh

#!/bin/sh

warn "crypt-shutdown: START"

warn "crypt-shutdown: systemctl status udev.service"
systemctl status udev.service 2>&1

warn "crypt-shutdown: dmsetup ls --target crypt"
dmsetup ls --target crypt 2>&1

warn "crypt-shutdown: lsblkl"
lsblk 2>&1

warn "crypt-shutdown: df -h"
df -h 2>&1

warn "crypt-shutdown: mount"
mount 2>&1

warn "crypt-shutdown: vgdisplay"
vgdisplay 2>&1

# Mark crypt devices for deferred removal.
# The dm module removes holding devices, so
# that the encryption keys can be released.
dmsetup ls --target crypt | while read -r name _; do
    if ! type "cryptsetup" > /dev/null 2>&1; then
        warn "cryptsetup not installed, skipping closing of encrypted devices"
        return
    fi

    warn "crypt-shutdown: lsof /dev/mapper/$name"
    lsof "/dev/mapper/$name" 2>&1

    warn "crypt-shutdown: findmnt /dev/mapper/$name"
    findmnt "/dev/mapper/$name" 2>&1

    warn "crypt-shutdown: cryptsetup status $name"
    cryptsetup status "$name" 2>&1

    warn "crypt-shutdown: cryptsetup --debug close $name"
    cryptsetup --debug close "$name" 2>&1

    warn "crypt-shutdown: cryptsetup status $name"
    cryptsetup status "$name" 2>&1

    warn "crypt-shutdown: cryptsetup --debug close $name --deferred"
    cryptsetup --debug close "$name" --deferred 2>&1

    warn "crypt-shutdown.s: name: $name - done"
done

warn "crypt-shutdown.s: END"

So there is really nothing still mounted. The issue is Udev cookie 0xd4dd417 (semid 3) waiting for zero. Thoughts?

[DanWin]

The latest commit should fix above mentioned udev issues. At this stage, udev has already been shut down, but may have left the control socket behind. Cryptsetup makes use of dmsetup, which in turn checks whether this socket exists. When dmsetup finds this socket file, it will try to synchronize with udev. But since udev is not running, it will wait forever. Simply removing the leftover socket file fixed it for me.

[adrelanos]

Great! Works for me. Thank you!

[adrelanos]

Could this be reviewed please?

[LaszloGombos]

Cryptsetup may not be installed on every system, if systemd-cryptsetup is used. That's why I made this feature optional in order not to break any running systems.

I better approach would be to call systemd-cryptsetup detach if systemd-cryptsetup is installed but cryptsetup is not installed. Is systemd-cryptsetup detach already called ? If it is than perhaps inly install crypt-shutdown.sh hook if systemd-cryptsetup is not used.

[DanWin]

@LaszloGombos agreed, but I did not find any reference on a delayed detaching in systemd-cryptsetup as is available in cryptsetup. I just checked again and when attempting to detach, it would fail with Failed to deactivate: Device or resource busy given that it's still used by device mapper, which is disassembled later on (but also removes the crypt device). So if we want to make use of systemd-cryptsetup I currently see two options:

1. Implement delayed detach in systemd-cryptsetup, which depends on upstream systemd accepting this change
2. Combine shutdown of dm and crypt modules in one script to check during dm shutdown if a target is of type crypt and remove encryption keys in that case.

[LaszloGombos]

Thanks.

I would recommend to try the following first

Implement delayed detach in systemd-cryptsetup, which depends on upstream systemd accepting this change

[DanWin]

I've created a pull request in systemd that would implement deferred detaching, see https://github.com/systemd/systemd/pull/32650 and the latest commit makes use of this new feature, if cryptsetup is not included in the initramfs.