ENOKEY when operating on encrypted disk

[ticpu]

Either from a freshly formatted FS or old, bcachefs fails to mount or use fsck -k in some conditions.

ENOKEY issue will most often appear when working from a sudo or Xorg session. When using a root TTY, it seems to be less likely, but not impossible, to reproduce. Rebooting and using a TTY to mount from just after startup has been a reliable way to mount my FS so far.

Userspace tools are not affected.

Here are some examples:

❯ bcachefs set-passphrase /dev/vgP4/bcache3
bch2_fs_open() bch_fs_open err opening /dev/vgP4/bcache3: insufficient_devices_to_start
Error opening /dev/vgP4/bcache3: insufficient_devices_to_start

❯ bcachefs set-passphrase /dev/vgP4/bcache*
Enter new passphrase: 
Enter same passphrase again: 

Return code was 0 there and normal mount didn't work anymore, as expected.

❯ bcachefs unlock /dev/vgP4/bcache*
Too many arguments

❯ bcachefs unlock /dev/vgP4/bcache1
Enter passphrase: 
(return 0)

I also tried with all disks one by one, then:

❯  /usr/bin/mount -t bcachefs $(ls -1 /dev/vgP4/bcache* | paste -sd:) /mnt/bcachefs
aoû 15 18:07:18 p4 kernel: bcachefs (d0337b38-cbe7-4c3c-90f7-4305fbe05171): error requesting encryption key: ENOKEY
aoû 15 18:07:18 p4 kernel: bcachefs (d0337b38-cbe7-4c3c-90f7-4305fbe05171): shutdown complete
aoû 15 18:07:18 p4 sh[3324455]: Error: Required key not available
aoû 15 18:07:18 p4 kernel: bcachefs: bch2_fs_get_tree() error: ENOKEY

❯ bcachefs mount -k ask UUID=d0337b38-cbe7-4c3c-90f7-4305fbe05171 /mnt/bcachefs
🔐 Enter passphrase:  •••••••••••••••         
Error: Required key not available

❯ dmesg |tail
[636148.916731] bcachefs (d0337b38-cbe7-4c3c-90f7-4305fbe05171): shutdown complete
[636242.195626] bcachefs (d0337b38-cbe7-4c3c-90f7-4305fbe05171): error requesting encryption key: ENOKEY
[636242.195629] bcachefs (d0337b38-cbe7-4c3c-90f7-4305fbe05171): shutdown complete
[636242.261278] bcachefs: bch2_fs_get_tree() error: ENOKEY

So things don't want to mount at all for now, however;

❯ bcachefs set-passphrase /dev/vgP4/bcache*
Enter passphrase: 
Enter passphrase: 
Enter new passphrase: 
Enter same passphrase again:

❯ bcachefs set-passphrase /dev/vgP4/bcache*
Enter passphrase: 
incorrect passphrase

Changing the passphrase works, using empty passphrase works as well but doesn't mount and still ask for passphrase if I change it again unlike the first time. I can change password as much as I want and it also recognize wrong passwords while using set-passphrase.

Oddly, it also asks for the old passphrase twice.

bcachefs fsck also fails after entering the right passphrase.

❯ bcachefs fsck /dev/vgP4/bcache*
Enter passphrase: 
fsck binary is version 1.10: disk_accounting_v3 but filesystem is 1.9: disk_accounting_v2 and kernel is 1.9: disk_accounting_v2, using kernel fsck
Running in-kernel offline fsck
bcachefs (d0337b38-cbe7-4c3c-90f7-4305fbe05171): error requesting encryption key: ENOKEY
bcachefs (d0337b38-cbe7-4c3c-90f7-4305fbe05171): shutdown complete

❯ bcachefs fsck /dev/vgP4/bcache*
Enter passphrase: 
incorrect passphrase

[ticpu]

After further investigation, the issue isn't completely isolated to a specific commit in the tools. It is possible to reproduce the same issue after running kernel 6.11-rc1 for a while and unmounting/re-mounting with v1.9.4 of the tools. I get ENOKEY and no way to mount until reboot, even after keyctl clear @u.

[ticpu]

After more digging, using a systemd service can't unlock the disk on Arch. Using but when "/bin/login" is used to authenticate first, the PAM modules that are loaded from there allows unlocking the FS.

[g2p]

Another failure mode to be aware of is with screen/tmux: the key could be scoped to a keyring attached to the first ssh connection; after reconnecting it becomes inaccessible, but still present enough to gum up the works. Workaround is:

keyctl new_session
keyctl link @u @s # optional?

(explanation found on Stackoverflow)