Cryptography Setup for luks container fails on 15.4

[phillxnet]

On a Leap 15.3 & Tumbleweed base we have successful systemd activation (on boot) of our Web-UI created LUKS containers setup to use the recommended native keyfile option. However on a Leap 15.4 base similarly configured LUKS container are not auto unlocked by system with the following per-container systemd service fail report:

rleap15-4:~ # systemctl status systemd-cryptsetup\@luks\\x2df35f8af4\\x2d288a\\x2d4380\\x2da6f7\\x2d7d2eddf16ae3.service 
× systemd-cryptsetup@luks\x2df35f8af4\x2d288a\x2d4380\x2da6f7\x2d7d2eddf16ae3.service - Cryptography Setup for luks-f35f8af4-288a-4380-a6f7-7d2eddf16ae3
     Loaded: loaded (/etc/crypttab; generated)
     Active: failed (Result: exit-code) since Mon 2023-05-01 18:20:19 WEST; 32min ago
       Docs: man:crypttab(5)
             man:systemd-cryptsetup-generator(8)
             man:systemd-cryptsetup@.service(8)
   Main PID: 602 (code=exited, status=1/FAILURE)

May 01 18:20:10 rleap15-4 systemd[1]: Starting Cryptography Setup for luks-f35f8af4-288a-4380-a6f7-7d2eddf16ae3...
May 01 18:20:10 rleap15-4 systemd-cryptsetup[602]: Set cipher aes, mode xts-plain64, key size 512 bits for device /dev/disk/by-uuid/f35f8af4-288a-4380-a6f7-7d2eddf16ae3.
May 01 18:20:19 rleap15-4 systemd-cryptsetup[602]: device-mapper: reload ioctl on   failed: Invalid argument
May 01 18:20:19 rleap15-4 systemd-cryptsetup[602]: Failed to activate with key file '/root/keyfile-f35f8af4-288a-4380-a6f7-7d2eddf16ae3': No such device
May 01 18:20:19 rleap15-4 systemd[1]: systemd-cryptsetup@luks\x2df35f8af4\x2d288a\x2d4380\x2da6f7\x2d7d2eddf16ae3.service: Main process exited, code=exited, status=1/FAILURE
May 01 18:20:19 rleap15-4 systemd[1]: systemd-cryptsetup@luks\x2df35f8af4\x2d288a\x2d4380\x2da6f7\x2d7d2eddf16ae3.service: Failed with result 'exit-code'.
May 01 18:20:19 rleap15-4 systemd[1]: Failed to start Cryptography Setup for luks-f35f8af4-288a-4380-a6f7-7d2eddf16ae3.

But the uuid matches the currently locked container and the indicated keyfile does exist:

rleap15-4:~ # ls -la /root/keyfile-f35f8af4-288a-4380-a6f7-7d2eddf16ae3
-rw------- 1 root root 2048 May  1 18:16 /root/keyfile-f35f8af4-288a-4380-a6f7-7d2eddf16ae3

Also the indicated drive

device /dev/disk/by-uuid/f35f8af4-288a-4380-a6f7-7d2eddf16ae3. resolves to the correct device:

rleap15-4:~ # ls -la /dev/disk/by-uuid/f35f8af4-288a-4380-a6f7-7d2eddf16ae3 lrwxrwxrwx 1 root root 9 May 1 18:20 /dev/disk/by-uuid/f35f8af4-288a-4380-a6f7-7d2eddf16ae3 -> ../../vda

which matches the Web-UI's info:

rleap15-4:~ # ls -la /dev/disk/by-id/virtio-0002
lrwxrwxrwx 1 root root 9 May  1 18:20 /dev/disk/by-id/virtio-0002 -> ../../vda

and:

rleap15-4:~ # cryptsetup luksUUID /dev/disk/by-id/virtio-0002
f35f8af4-288a-4380-a6f7-7d2eddf16ae3

[phillxnet]

Post keyfile setup in TW (and reboot) we have the expected unlocked LUKS container and the consequent open luks volume as a result, and as part of a four (all LUKS encrypted) member raid6-1c4 pool.

On first thought, this may just be a difference in the systemd config between the OS variants re compliance with our keyfile location:

/root

That may be why the Leap variant appears not to be able to see our keyfiles.

[phillxnet]

The systemd services used to auto-open (via keyfile) are located in:

rleap15-4:/var/run/systemd/generator

and an example contents is:

cat systemd-cryptsetup\@luks\x2df35f8af4\x2d288a\x2d4380\x2da6f7\x2d7d2eddf16ae3.service

# Automatically generated by systemd-cryptsetup-generator

[Unit]
Description=Cryptography Setup for %I
Documentation=man:crypttab(5) man:systemd-cryptsetup-generator(8) man:systemd-cryptsetup@.service(8)
SourcePath=/etc/crypttab
DefaultDependencies=no
IgnoreOnIsolate=true
After=cryptsetup-pre.target systemd-udevd-kernel.socket
Before=blockdev@dev-mapper-%i.target
Wants=blockdev@dev-mapper-%i.target
Conflicts=umount.target
Before=cryptsetup.target
RequiresMountsFor=/root/keyfile-f35f8af4-288a-4380-a6f7-7d2eddf16ae3
BindsTo=dev-disk-by\x2duuid-f35f8af4\x2d288a\x2d4380\x2da6f7\x2d7d2eddf16ae3.device
After=dev-disk-by\x2duuid-f35f8af4\x2d288a\x2d4380\x2da6f7\x2d7d2eddf16ae3.device
Before=umount.target

[Service]
Type=oneshot
RemainAfterExit=yes
TimeoutSec=0
KeyringMode=shared
OOMScoreAdjust=500
ExecStart=/usr/lib/systemd/systemd-cryptsetup attach 'luks-f35f8af4-288a-4380-a6f7-7d2eddf16ae3' '/dev/disk/by-uuid/f35f8af4-288a-4380-a6f7-7d2eddf16ae3' '/root/keyfile-f35f8af4-288a-4380-a6f7-7d2eddf16ae3' 'luks'
ExecStop=/usr/lib/systemd/systemd-cryptsetup detach 'luks-f35f8af4-288a-4380-a6f7-7d2eddf16ae3'

With the TW variant looking the same bar uuid references.

[phillxnet]

Successful Tumbleweed keyfile unlock on boot:

buildvm:~ # systemctl status systemd-cryptsetup@luks\\x2d877e21c8\\x2df274\\x2d4e58\\x2d877a\\x2de091c50cf7f2.service 
● systemd-cryptsetup@luks\x2d877e21c8\x2df274\x2d4e58\x2d877a\x2de091c50cf7f2.service - Cryptography Setup for luks-877e21c8-f274-4e58-877a-e091c50cf7f2
     Loaded: loaded (/etc/crypttab; generated)
     Active: active (exited) since Tue 2023-05-02 17:01:27 WEST; 48min ago
       Docs: man:crypttab(5)
             man:systemd-cryptsetup-generator(8)
             man:systemd-cryptsetup@.service(8)
   Main PID: 580 (code=exited, status=0/SUCCESS)
        CPU: 7.833s

May 02 17:01:16 buildvm systemd[1]: Starting Cryptography Setup for luks-877e21c8-f274-4e58-877a-e091c50cf7f2...
May 02 17:01:16 buildvm systemd-cryptsetup[580]: Set cipher aes, mode xts-plain64, key size 512 bits for device /dev/disk/by-uuid/877e21c8-f274-4e58-877a-e091c50cf7f2.
May 02 17:01:27 buildvm systemd[1]: Finished Cryptography Setup for luks-877e21c8-f274-4e58-877a-e091c50cf7f2.

[phillxnet]

Systemd versions

[EDIT to add 15.3]

rleap15-3:~ # rpm -qi systemd | grep Version Version : 246.16

15.4

rleap15-4:~ # rpm -qi systemd | grep Version Version : 249.16

Tumbleweed

buildvm:~ # rpm -qi systemd | grep Version Version : 253.3

[phillxnet]

Noting the wrapper we use to reset device specific systemd-cryptsetup@ services: https://github.com/rockstor/rockstor-core/blob/ac68e65e7eb9cb676052ce93bc4e5e8bf85e8e56/src/rockstor/system/osi.py#L2345-L2360

[phillxnet]

Our "Build on openSUSE" Leap 15.3 also looks to be unlocking LUKS containers configured to use keyfiles via the Web-UI just fine also. So this issue looks to concern our Leap 15.4 base only.

[phillxnet]

Context of systemd services:

rleap15-3:~ # systemctl list-unit-files |grep crypt
systemd-cryptsetup@luks\x2d499c89f6\x2d4c5f\x2d4aa7\x2d9eca\x2dfa5691345349.service generated       -            
systemd-cryptsetup@luks\x2d8378b01c\x2de694\x2d4c46\x2d9bda\x2d99d1bd62c49e.service generated       -            
systemd-cryptsetup@luks\x2dbe52453d\x2da197\x2d4d6f\x2dbd98\x2d2823ac68bfc6.service generated       -            
system-systemd\x2dcryptsetup.slice                                                  static          -            
cryptsetup-pre.target                                                               static          -            
cryptsetup.target                                                                   static          -            
remote-cryptsetup.target                                                            disabled        disabled

[phillxnet]

Upstream: https://github.com/openSUSE/systemd Upstream wiki: https://www.freedesktop.org/wiki/Software/systemd/ Downsteam to openSUSE (but only last 2 versions v252.9 & v253.3): https://github.com/openSUSE/systemd OBS Factory: https://build.opensuse.org/package/show/Base:System/systemd

[phillxnet]

Linking to an issue that is suspected as related in upstream re systemd version: https://github.com/systemd/systemd/issues/19193

[phillxnet]

This issue should be closed upon EOL for the target OS: Leap 15.4. Expected to be end of November 2023. Ideally after we have a Stable Release Candidate out for Leap 15.5.