booting with bcachefs compression on is broken

[codebam]

Hey I'm using NixOS, and I can't boot after updating to 1.7.0.

I use both encryption and compression.

This is my configuration: https://github.com/codebam/nixos (I've pinned 1.4.0 for the time being).

# configuration.nix
  boot = {
    loader = {
      systemd-boot.enable = true;
      efi.canTouchEfiVariables = true;
      systemd-boot.configurationLimit = 10;
    };
    kernelPackages = pkgs.linuxPackages_testing;
    supportedFilesystems = [ "bcachefs" ];
  };
# hardware-configuration.nix
  fileSystems."/" =
    {
      device = "UUID=2ef09b4b-9fb1-4815-8dd2-4aede2ae9add";
      fsType = "bcachefs";
      options = [ "compression=lz4" "background_compression=zstd" ];
    };

What I've tried is here: https://github.com/NixOS/nixpkgs/issues/309388

[koverstreet]

I can't do anything without logs.

[JohnRTitor]

How to get the logs, except capturing the boot screen's image? The system does not boot, so journalctl -b 1 isn't helpful.

[koverstreet]

Capture the boot image's screen, use a camera if you have to

[codebam]

Here is a video in HEVC format https://r2.seanbehan.ca/b8a1e

Relevant frame: https://r2.seanbehan.ca/8208d

Edit: in H264: https://r2.seanbehan.ca/a1953

[chaosbiber]

Hi, I've tried to increase verbosity but with not much luck, here's my transcript:

unlocking successful.
mounting UUID=1bb5d54a-xxxx-xxxx-xxxx-xxxxxxxxxx on /...
(block of raid6 algorithm speeds)
[  53.2xxxx] xor: automatically using best checksumming function  avx
[  53.3xxxx] bcachefs (UUID=1bb5d54a-xxxx-xxxx-xxxxxxxxxx): error reading superblock: (null)
ERROR - bcachefs::commands::cmd_mount: Fatal error: No such device or address (os error 6)
(errors about reading default superblock messages)
mount: mounting UUID=1bb5d54a-xxxx-xxxx-xxxx-xxxxxxxxxx on /mnt-root/ failed: No such file or directory

(nixos, flake update included new kernel 6.8.8->6.8.9 and bcachefs-tools 1.41->1.70; rootfs with enabled encryption)

[tmuehlbacher]

I have the same or at least a similar problem with two of my systems, also from the nixpkgs upgrade of bcachefs-tools from 1.4.1 to 1.7.0. I also have encryption enabled but not compression. I think it might have something to do with the version of the superblock?

This is the superblock from one system with a single NVMe, the other system has two NVMes with 1 data replica.

``` External UUID: 314767f3-de4e-4d10-8e38-6b0ba123313e Internal UUID: 2592e602-e831-4a86-bf58-b0fa1c1bafe0 Magic number: c68573f6-66ce-90a9-d96a-60cf803df7ef Device index: 0 Label: nix Version: 1.4: member_seq Version upgrade complete: 1.4: member_seq Oldest version on disk: 1.3: rebalance_work Created: Sat Jan 20 12:37:05 2024 Sequence number: 348 Time of last write: Mon May 6 19:22:48 2024 Superblock size: 4680 Clean: 0 Devices: 1 Sections: members_v1,crypt,replicas_v0,disk_groups,clean,journal_v2,counters,members_v2,errors,ext,downgrade Features: reflink,new_siphash,inline_data,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,reflink_inline_data,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes Compat features: alloc_info,alloc_metadata,extents_above_btree_updates_done,bformat_overflow_done Options: block_size: 512 B btree_node_size: 256 KiB errors: continue [ro] panic metadata_replicas: 1 data_replicas: 1 metadata_replicas_required: 1 data_replicas_required: 1 encoded_extent_max: 64.0 KiB metadata_checksum: none [crc32c] crc64 xxhash data_checksum: none [crc32c] crc64 xxhash compression: none background_compression: none str_hash: crc32c crc64 [siphash] metadata_target: none foreground_target: none background_target: none promote_target: none erasure_code: 0 inodes_32bit: 1 shard_inode_numbers: 1 inodes_use_key_cache: 1 gc_reserve_percent: 8 gc_reserve_bytes: 0 B root_reserve_percent: 0 wide_macs: 0 acl: 1 usrquota: 0 grpquota: 0 prjquota: 0 journal_flush_delay: 1000 journal_flush_disabled: 0 journal_reclaim_delay: 100 journal_transaction_names: 1 version_upgrade: [compatible] incompatible none nocow: 0 members_v2 (size 152): Device: 0 Label: main (0) UUID: 92ca236e-1a61-4edc-8868-e494deec18aa Size: 3.72 TiB read errors: 0 write errors: 0 checksum errors: 0 seqread iops: 0 seqwrite iops: 0 randread iops: 0 randwrite iops: 0 Bucket size: 512 KiB First bucket: 0 Buckets: 7811986 Last mount: Mon May 6 19:22:48 2024 Last superblock write: 348 State: rw Data allowed: journal,btree,user Has data: journal,btree,user Durability: 1 Discard: 1 Freespace initialized: 1 errors (size 8): ```

However, I also had an added problem where I also could not boot anymore with bcachefs-tools 1.4.1 when I use any of the 6.9-rc kernels. Kernel 6.8.9 works fine still, but the 6.9 wouldn't mount with a message like this:

... error reading superblock: error opening UUID=...: ENOENT

That just magically fixed itself on one of the systems now after trying again to get the exact phrasing right on this error message with 6.9-rc7. Previously this had even caused an error to be recorded in the superblock, fs_usage_nr_inodes_wrong.

The boot problem with bcachefs-tools 1.7.0 persists however, even though I am now using linux 6.9-rc7 and the superblock has been upgraded to bcachefs 1.7.

A weird thing is that I can not reproduce this anymore with a fresh installation where the rootfs was created with bcachefs-tools 1.4.1. So only the two systems I created around the middle of January have these issues.

Sorry if my description is a bit chaotic. :sweat_smile:

[tasleson]

@chaosbiber @tmuehlbacher What are you using in your fstab? device node, UUID=? Also it sounds like mostly single block device file system, not multi-device right?

Update: Also what version of blkid are you using (check util-linux)

[chaosbiber]

What are you using in your fstab? device node, UUID=?

UUID=1bb5d54a-cafa-4012-9831-12f31cb8a8fa / bcachefs x-initrd.mount,compression=lz4 0 0

generated from nixos-config

fileSystems."/" =
{ device = "UUID=...";
  fsType = "bcachefs";
  options = [ "compression=lz4" ];
};

Also it sounds like mostly single block device file system, not multi-device right?

correct

Update: Also what version of blkid are you using (check util-linux)

blkid from util-linux 2.39.3 (libblkid 2.39.3, 04-Dec-2023) (did not change during the update)

show-super

``` External UUID: 1bb5d54a-cafa-4012-9831-12f31cb8a8fa Internal UUID: 6a8c9cfa-4c2d-4f91-a50d-15b0a32016ad Magic number: c68573f6-66ce-90a9-d96a-60cf803df7ef Device index: 0 Label: nixos Version: 1.4: member_seq Version upgrade complete: 1.4: member_seq Oldest version on disk: 1.3: rebalance_work Created: Mon Feb 26 20:26:49 2024 Sequence number: 351 Time of last write: Mon May 6 22:08:32 2024 Superblock size: 4632 Clean: 0 Devices: 1 Sections: members_v1,crypt,replicas_v0,clean,journal_seq_blacklist,journal_v2,counters,members_v2,errors,ext,downgrade Features: lz4,journal_seq_blacklist_v3,reflink,new_siphash,inline_data,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,reflink_inline_data,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes Compat features: alloc_info,alloc_metadata,extents_above_btree_updates_done,bformat_overflow_done Options: block_size: 512 B btree_node_size: 256 KiB errors: continue [ro] panic metadata_replicas: 1 data_replicas: 1 metadata_replicas_required: 1 data_replicas_required: 1 encoded_extent_max: 64.0 KiB metadata_checksum: none [crc32c] crc64 xxhash data_checksum: none [crc32c] crc64 xxhash compression: lz4 background_compression: none str_hash: crc32c crc64 [siphash] metadata_target: none foreground_target: none background_target: none promote_target: none erasure_code: 0 inodes_32bit: 1 shard_inode_numbers: 1 inodes_use_key_cache: 1 gc_reserve_percent: 8 gc_reserve_bytes: 0 B root_reserve_percent: 0 wide_macs: 0 acl: 1 usrquota: 0 grpquota: 0 prjquota: 0 journal_flush_delay: 1000 journal_flush_disabled: 0 journal_reclaim_delay: 100 journal_transaction_names: 1 version_upgrade: [compatible] incompatible none nocow: 0 members_v2 (size 144): Device: 0 Label: (none) UUID: 64e3c6c4-c35b-4066-ae11-33cf7f93ceb6 Size: 465 GiB read errors: 0 write errors: 0 checksum errors: 0 seqread iops: 0 seqwrite iops: 0 randread iops: 0 randwrite iops: 0 Bucket size: 256 KiB First bucket: 0 Buckets: 1905808 Last mount: Mon May 6 22:08:32 2024 Last superblock write: 351 State: rw Data allowed: journal,btree,user Has data: journal,btree,user Durability: 1 Discard: 0 Freespace initialized: 1 errors (size 8): ```

[tmuehlbacher]

This is my fstab line, generated by NixOS:

UUID=314767f3-de4e-4d10-8e38-6b0ba123313e /nix bcachefs x-initrd.mount,defaults,lazytime 0 0

blkid --version:

blkid from util-linux 2.39.3  (libblkid 2.39.3, 04-Dec-2023)

Actually, one of my systems is multi-device. But both have the same problem. The super block info for the single-device fs is already in my previous comment. Here is the super block info for the multi-device fs:

``` External UUID: 314767f3-de4e-4d10-8e38-6b0ba123313e Internal UUID: 4e882fdd-3db0-49c1-806e-0717ccfdbd75 Magic number: c68573f6-66ce-90a9-d96a-60cf803df7ef Device index: 0 Label: nix Version: 1.7: (unknown version) Version upgrade complete: 1.7: (unknown version) Oldest version on disk: 1.3: rebalance_work Created: Sun Jan 21 19:05:14 2024 Sequence number: 11470 Time of last write: Mon May 6 19:41:04 2024 Superblock size: 5776 Clean: 0 Devices: 2 Sections: members_v1,crypt,replicas_v0,disk_groups,clean,journal_seq_blacklist,journal_v2,counters,members_v2,errors,ext,downgrade Features: journal_seq_blacklist_v3,reflink,new_siphash,inline_data,new_extent_overwrite,btree_ptr_v2,extents_above_btree_updates,btree_updates_journalled,reflink_inline_data,new_varint,journal_no_flush,alloc_v2,extents_across_btree_nodes Compat features: alloc_info,alloc_metadata,extents_above_btree_updates_done,bformat_overflow_done Options: block_size: 512 B btree_node_size: 256 KiB errors: continue [ro] panic metadata_replicas: 2 data_replicas: 1 metadata_replicas_required: 1 data_replicas_required: 1 encoded_extent_max: 64.0 KiB metadata_checksum: none [crc32c] crc64 xxhash data_checksum: none [crc32c] crc64 xxhash compression: none background_compression: none str_hash: crc32c crc64 [siphash] metadata_target: none foreground_target: none background_target: none promote_target: none erasure_code: 0 inodes_32bit: 1 shard_inode_numbers: 1 inodes_use_key_cache: 1 gc_reserve_percent: 8 gc_reserve_bytes: 0 B root_reserve_percent: 0 wide_macs: 0 acl: 1 usrquota: 0 grpquota: 0 prjquota: 0 journal_flush_delay: 1000 journal_flush_disabled: 0 journal_reclaim_delay: 100 journal_transaction_names: 1 version_upgrade: [compatible] incompatible none nocow: 0 members_v2 (size 288): Device: 0 Label: main (0) UUID: e793ab38-a95a-4bef-b4fe-6b64fe8de7f5 Size: 1.86 TiB read errors: 0 write errors: 0 checksum errors: 0 seqread iops: 0 seqwrite iops: 0 randread iops: 0 randwrite iops: 0 Bucket size: 512 KiB First bucket: 0 Buckets: 3904978 Last mount: Mon May 6 19:41:04 2024 Last superblock write: 11470 State: rw Data allowed: journal,btree,user Has data: journal,btree,user Durability: 1 Discard: 1 Freespace initialized: 1 Device: 1 Label: extra (1) UUID: 8d9a2ebc-a518-4a05-986c-d0c43338b35a Size: 1.82 TiB read errors: 0 write errors: 0 checksum errors: 0 seqread iops: 0 seqwrite iops: 0 randread iops: 0 randwrite iops: 0 Bucket size: 1.00 MiB First bucket: 0 Buckets: 1907728 Last mount: Mon May 6 19:41:04 2024 Last superblock write: 11470 State: rw Data allowed: journal,btree,user Has data: journal,btree,user Durability: 1 Discard: 1 Freespace initialized: 1 errors (size 24): fs_usage_nr_inodes_wrong 1 Thu Mar 28 19:57:18 2024 ```

[koverstreet]

Can you try 'bcachefs mount' with -v?

[tmuehlbacher]

Here is the terminal output from a separate live system, using the same nixpkgs revisions as the actual systems.

In the first command it didn't ask for a password because the key was still in the kernel keyring from a prior mount.

It works here in a full system, just not in the initramfs, apparently.

``` $ bcachefs version 1.4.1 $ sudo bcachefs mount -v UUID=314767f3-de4e-4d10-8e38-6b0ba123313e mnt DEBUG - bcachefs_rust::cmd_mount: enumerating devices with UUID 314767f3-de4e-4d10-8e38-6b0ba123313e DEBUG - bcachefs_rust::cmd_mount: enumerating udev devices INFO - bcachefs_rust::key: checking if key exists for filesystem 314767f3-de4e-4d10-8e38-6b0ba123313e INFO - bcachefs_rust::key: Key has became available INFO - bcachefs_rust::cmd_mount: mounting with params: device: /dev/nvme1n1p2:/dev/nvme0n1p1, target: mnt, options: DEBUG - bcachefs_rust::cmd_mount: parsing mount options: INFO - bcachefs_rust::cmd_mount: mounting bcachefs filesystem, mnt INFO - bcachefs_rust::cmd_mount: mounting filesystem INFO - bcachefs_rust::cmd_mount: Successfully mounted $ sudo umount mnt $ nix shell github:nixos/nixpkgs/nixos-unstable#bcachefs-tools $ bcachefs version 1.7.0 $ sudo bcachefs mount -v UUID=314767f3-de4e-4d10-8e38-6b0ba123313e mnt DEBUG - bcachefs::commands::cmd_mount: enumerating devices with UUID 314767f3-de4e-4d10-8e38-6b0ba123313e DEBUG - bcachefs::commands::cmd_mount: enumerating udev devices bcachefs (/dev/nvme1n1): error reading default superblock: Not a bcachefs superblock (got magic 00000000-0000-0000-0000-000000000000) bcachefs (/dev/nvme1n1): error reading superblock: Not a bcachefs superblock layout bcachefs (/dev/nvme1n1p1): error reading default superblock: Not a bcachefs superblock (got magic 00000000-0000-0000-0000-000000000000) bcachefs (/dev/nvme1n1p1): error reading superblock: Not a bcachefs superblock layout bcachefs (/dev/sda): error reading default superblock: Not a bcachefs superblock (got magic 00000000-0000-0000-0000-000000000000) bcachefs (/dev/sda): error reading superblock: Not a bcachefs superblock layout bcachefs (/dev/sda1): error reading default superblock: Not a bcachefs superblock (got magic 00000000-0000-0000-0000-000000000000) bcachefs (/dev/sda1): error reading superblock: Not a bcachefs superblock layout bcachefs (/dev/nvme0n1): error reading default superblock: Not a bcachefs superblock (got magic 00000000-0000-0000-0000-000000000000) bcachefs (/dev/nvme0n1): error reading superblock: Not a bcachefs superblock layout bcachefs (/dev/loop0): error reading default superblock: IO error: -5 bcachefs (/dev/loop0): error reading superblock: IO error: -5 bcachefs (/dev/loop1): error reading default superblock: IO error: -5 bcachefs (/dev/loop1): error reading superblock: IO error: -5 bcachefs (/dev/loop2): error reading default superblock: IO error: -5 bcachefs (/dev/loop2): error reading superblock: IO error: -5 bcachefs (/dev/loop3): error reading default superblock: IO error: -5 bcachefs (/dev/loop3): error reading superblock: IO error: -5 bcachefs (/dev/loop4): error reading default superblock: IO error: -5 bcachefs (/dev/loop4): error reading superblock: IO error: -5 bcachefs (/dev/loop5): error reading default superblock: IO error: -5 bcachefs (/dev/loop5): error reading superblock: IO error: -5 bcachefs (/dev/loop6): error reading default superblock: IO error: -5 bcachefs (/dev/loop6): error reading superblock: IO error: -5 bcachefs (/dev/loop7): error reading default superblock: IO error: -5 bcachefs (/dev/loop7): error reading superblock: IO error: -5 bcachefs (/dev/zram0): error reading default superblock: Not a bcachefs superblock (got magic 00000000-0000-0000-0000-000000000000) bcachefs (/dev/zram0): error reading superblock: IO error: -5 INFO - bcachefs::key: Attempting to unlock master key for filesystem 314767f3-de4e-4d10-8e38-6b0ba123313e, using unlock policy Ask Enter passphrase: INFO - bcachefs::key: Key has become available INFO - bcachefs::commands::cmd_mount: mounting with params: device: /dev/nvme1n1p2:/dev/nvme0n1p1, target: mnt, options: DEBUG - bcachefs::commands::cmd_mount: parsing mount options: INFO - bcachefs::commands::cmd_mount: mounting bcachefs filesystem, mnt INFO - bcachefs::commands::cmd_mount: mounting filesystem INFO - bcachefs::commands::cmd_mount: Successfully mounted $ sudo umount mnt ```

[koverstreet]

So that looks like a keyring issue

[chaosbiber]

I've created a nix installer based on nix-unstable so it should use most of the same package versions. Ran it on the system where booting fails. If I haven't misspelled my passphrase in all 6 or 7 tries, then yes, might look like that password doesn't reach its destination.

Edit: using kernel 6.8.9 with bcachefs-tools 1.7.0

Sorry for the dust, it's on my laptop, not your displays...

[chaosbiber]

Deleted because I just reproduced the issue mentioned also in the nixos wiki.

[codebam]

@chaosbiber I can confirm I have the same issue. If you mount with bcachefs unlock -k session /dev/sda it works.

[chaosbiber]

@codebam Ah, right, I just reproduced an older issue for the 23.11 installer. But using the 1.7.0 again: with both the unlock -k session and the keyctl link @u @s plus unlock commands I can mount the volume, but it asks for the passphrase twice (unlock and mount)!

[reedriley]

I can confirm hitting this issue trying to upgrade bcachefs-tools to 1.6.x a month ago on NixOS. bcachefs-tools have always worked perfectly fine for me when run interactively from a rescue USB. (NixOS skipped bcachefs-tools 1.6x entirely; so if anyone's looking for root causes they're probably further back in time.)

[reedriley]

If I were to be suspicious of a particular commit here in bcachefs-tools, it'd probably be: https://github.com/koverstreet/bcachefs-tools/commit/0a284fc4ffcbb46f0a4b921415ef12a9c75fa05c (and possibly, subsequent changes to mount.rs)?

Could some kind of subtle "bug" have been introduced with the conversion of main to Rust? For example - something subtle like changing the return code when executed using the mount.bcachefs symlink on a successful mount would be enough to break NixOS's stage1 script (I believe).

[JohnRTitor]

(NixOS skipped bcachefs-tools 1.6x entirely; so if anyone's looking for root causes they're probably further back in time.)

If you are interested can probably run git bisect to find the actual commit that did this. This repo also provides a flake, so you can easily update to a intermediate version like 1.5 or 1.6 using a specific tag as flake input. Then overlay the package on top of system bcachefs-tools.

[reedriley]

If I do run a bisect - do y'all consider it safe to use intermediate versions? Or should I restrict to tagged releases?

[JohnRTitor]

I'd recommend staying on the safe side and test tagged releases first, then we can move from there. @koverstreet can confirm better though if it's safe to use intermediate commits.

[reedriley]

Alright; ran a bisect on my machine (restricting to tagged releases). The last good release was v1.6.3 and the first bad release is v1.6.4.

git log --bisect --oneline --decorate --graph:

• 010182e7 (tag: v1.6.4) v1.6.4
• 3388cf00 powerpc: use "long long" for 64-bit types in userspace
• 16661be2 bch2_sb_is_encrypted{,_and_locked}
• 817e9576 Apply naming convention: passphrase->unlock->key->decrypt->fs
• 387e2a01 Test 'fix'?
• 8a800c67 Replace 7 spaces and a tab with a tab.
• b582c9b4 Improve error message
• eaf55750 Rename sb to block_device
• ceb9eaa0 Rename password{,_policy}
• 37c0ae2e Rename KeyLocation to more appropriate KeyPolicy
• ceb259b2 (HEAD) Typo
• 46ed7246 Rename sbs to block_devices_to_mount
• b7c43996 Rephrase help message
• 911c5326 Remove unnecessary brackets
• d67643f2 Add decryption by key_file
• 0ff96b2a Add fmt::Display for KeyLocation
• 86049a16 Refactor ask_for_key to call new decrypt_master_key
• 0728677c Fix dirent test: lines have changed
• 466802d8 Fix line check
• 25e84a99 rust: use libc::Ioctl type for ioctl() request argument

I also added a little extra logging in stage-1-init.sh; and can tell you a little bit more precisely about what regressed:

• Calling mount "/mnt-root/persist" returns a failure status code (haven't yet been able to confirm if the mounted fs was actually present afterwards), and

  • /etc/fstab contains the line UUID=<insert bcachefs uuid> /mnt-root/persist bcachefs verbose,fix_errors (no doubt with different white space, I had to copy from a photo).

  If it's relevant - I'm not running compression but am running encryption; and have a multi-device setup.

[JohnRTitor]

@marcin-github would it be possible for you to test if you can reproduce this, on a NixOS machine? Not sure if it would be relevant in Fedora or Debian.

[marcin-github]

@marcin-github would it be possible for you to test if you can reproduce this, on a NixOS machine? Not sure if it would be relevant in Fedora or Debian.

Meseems you want to mention someone else :)

[pimeys]

Having the same issue. NixOS with Linux 6.8.9, bcachefs-tools 1.7.0, configuration:

fileSystems."/" = {
  device = "UUID=5f910790-3f93-4e9e-baf4-13b69719dc6a";
  fsType = "bcachefs";
  options = [
    "compression=lz4"
    "fix_errors=yes"
    "nojournal_transaction_names"
    "relatime"
    "discard"
    "background_compression=lz4"
  ];
};

After typing my password on boot, getting an error:

bcachefs (UUID=5f910790-3f93-4e9e-baf4-13b69719dc6a): error reading superblock: (null)
bcachefs::commands::cmd_mount: Fatal error: No such device or address (os error 6)

Edit: using encryption together with compression.

[JohnRTitor]

@marcin-github would it be possible for you to test if you can reproduce this, on a NixOS machine? Not sure if it would be relevant in Fedora or Debian.

Meseems you want to mention someone else :)

Well you seem to be quite active in this repo testing bcachefs along with @tasleson. This needs an urgent patch, as v1.7.0 is supposed to ship with the upcoming NixOS 24.05 release at the end of May.

[marcin-github]

Well, I'm not using nixos, I'm not using bcachefs as root fs and I'm not using enryption. What I can say is that -tools on latest commits works on my host. I'm not sure how can I help you and what do you expect from me :(

[reedriley]

Got confirmation from @koverstreet on IRC that he thinks it should be reasonably safe to bisect further between v1.6.3 and v1.6.4; will hopefully have time to do so later tonight.

@marcin-github: Sadly, the tools mostly work fine for me as well - just not in initramfs.

@JohnRTitor: What's the precise reason for wanting to ship bcachefs-tools-1.7.0 with NixOS 24.05? Is 24.05 shipping with a 6.9 kernel? If not - I'd actually suggest that bcachefs-tools-1.6.x (specifically 1.6.3 because I believe it works) would be more appropriate because it matches the disk format in the 6.8 kernel?

[JohnRTitor]

6.9 kernel release is due in a week, and 24.05 NixOS release will definitely ship with it (release schedule). The ISO's, by default, however will stick to 6.6 LTS releases, so this issue will not affect most of the users, except those who are actively using bcachefs filesystem and choose to upgrade their system.

6.9 kernel uses bcachefs version 1.7.0, so obviously it would have been better to ship bcachefs-tools 1.7.0 with it. The kernel can automatically downgrade bcachefs version to better match the system though, so I am not too worried about a version mismatch. As things stand, I'll probably downgrade the package to 1.6.3.

[codebam]

I tried building and booting some of the commits between v1.6.3 and v.1.6.4

• 86049a1641535f451fdd5a8bf885ecb925adbf1e doesn't boot
• 0728677cdc325d3f9ff37f6a665eca13af5e50cc boots
• 86049a1641535f451fdd5a8bf885ecb925adbf1e doesn't boot

So it looks like the issue is with 86049a1641535f451fdd5a8bf885ecb925adbf1e ?

[reedriley]

That diff doesn't have any obvious issues in it, unless it's the reordering of this check from before the password prompt to after it and inside of the decryption logic? Won't be able to try myself for at least several hours - but would be curious if moving it back to the start of ask_for_key resolves things...

https://github.com/koverstreet/bcachefs-tools/commit/86049a1641535f451fdd5a8bf885ecb925adbf1e#diff-0988d5d2259e1a401001675578c38d09c46290b877b66b07fb96a7e034ccdf70R95-R98

[codebam]

@reedriley Yes actually, moving it back does in fact boot 12164c97df4c936d137d556ea6a49d809163f66f

[tmuehlbacher]

Yeah, I believe it makes a lot of sense. Not sure about rpassword but stdin.read_line() will definitely block forever if nothing comes in on stdin. So we never reach the point of being able to check the kernel keyring with check_for_key()

[codebam]

After rebasing 12164c97df4c936d137d556ea6a49d809163f66f onto 477670f48167cac1b871b061713cc1b594a2a941 (https://github.com/codebam/bcachefs-tools/commits/try-fix/ , https://github.com/codebam/bcachefs-tools/commits/fix-nixos-stage1) The password prompt doesn't read in passwords properly. It prompts, but pressing <CR> twice puts you on a new line.

Edit: tried with @tmuehlbacher's commit as well and no dice, at least not on master

[tmuehlbacher]

I didn't really test that commit yet. I will have to look into how to best test this (i.e. is master currently safe to use or do I have to rebase onto 1.7.0 to test). If someone would be willing to try, that would also be cool but no pressure. 🙂

[tmuehlbacher]

@codebam thanks, so still some problems on master? I tested it by branching off from v1.7.0 and cherry-picking the commit from my PR and that boots now for me on kernel 6.9-rc7

[codebam]

@codebam thanks, so still some problems on master? I tested it by branching off from v1.7.0 and cherry-picking the commit from my PR and that boots now for me on kernel 6.9-rc7

Yeah but it's a different problem on master where you can't enter the passphrase. I'm on this commit with your commit and it boots fine 5531accc97da082b7a102240e34fdf15c68a8991 also 6.9-rc7

[tmuehlbacher]

I pushed another commit to #263 that makes mounting work on master now. :)

[JohnRTitor]

Please test the latest changes so that it is fixed in upstream. NixOS users can easily do so like https://github.com/JohnRTitor/nix-conf/commit/b60df8a18feb8c9e6e4edc16fb62fe2a5ad0449b

[codebam]

Tested and working on rc7