emergency read only during evacuate

[Lykos153]

I've since moved to using bcachefs as my rootfs and it's working well so far.

Until now - while evacuating a device, I just got

[77996.234990] bcachefs (677cf0a7-1abe-4ce3-876c-2ca63301229d): initializing freespace
[78023.404412] bcachefs (nvme0n1p4): ro
[78884.749247] bcachefs (sdh1): rw
[78890.993497] bcachefs (sdh1): ro
[90087.177656] bcachefs (677cf0a7-1abe-4ce3-876c-2ca63301229d): fatal error - emergency read only
[90087.177763] kernel BUG at fs/bcachefs/io_write.c:532!
[90087.177778] Workqueue: bcachefs bch2_write_point_do_index_updates [bcachefs]
[90087.177861] RIP: 0010:__bch2_write_index+0x281/0x290 [bcachefs]
[90087.177944]  ? __bch2_write_index+0x281/0x290 [bcachefs]
[90087.178002]  ? __bch2_write_index+0x281/0x290 [bcachefs]
[90087.178061]  ? __bch2_write_index+0x281/0x290 [bcachefs]
[90087.178118]  ? __bch2_write_index+0x281/0x290 [bcachefs]
[90087.178171]  bch2_write_point_do_index_updates+0xb1/0x160 [bcachefs]
[90087.178295]  uvcvideo snd_intel_dspcfg snd_intel_sdw_acpi uvc i2c_i801 ecdh_generic i2c_smbus gspca_vc032x snd_hda_codec snd_usbmidi_lib gspca_main snd_ump ledtrig_audio snd_hda_core nf_tables videobuf2_vmalloc snd_rawmidi videobuf2_memops snd_seq_device videobuf2_v4l2 mei_me lpc_ich snd_hwdep snd_pcm videodev mei snd_timer hid_jabra rfkill ecc sch_fq_codel crc16 snd videobuf2_common mc uinput atkbd libps2 serio vivaldi_fmap pl2303 soundcore loop joydev input_leds tiny_power_button cpufreq_ondemand led_class xt_nat mousedev nf_nat nf_conntrack evdev nf_defrag_ipv6 button nf_defrag_ipv4 br_netfilter mac_hid veth bridge stp llc tun kvm_intel kvm vfio_pci vfio_pci_core vfio_iommu_type1 vfio iommufd fuse efi_pstore configfs nfnetlink zram efivarfs dmi_sysfs ip_tables x_tables autofs4 poly1305_generic libpoly1305 poly1305_x86_64 chacha_generic chacha_x86_64 libchacha bcachefs libcrc32c crc32c_generic lz4_compress lz4hc_compress xor raid6_pq hid_generic usbhid hid sd_mod i915 nvme nvme_core i2c_algo_bit drm_buddy ttm ahci
[90087.795437] RIP: 0010:__bch2_write_index+0x281/0x290 [bcachefs]
[90095.761254] bcachefs (677cf0a7-1abe-4ce3-876c-2ca63301229d): bch2_btree_update_start(): error EIO
[90095.761274] bcachefs (677cf0a7-1abe-4ce3-876c-2ca63301229d inum 0 offset 1818873561088): move write error while doing btree update: EIO

The last two lines are then repeated over and over with different offsets.

I thought I might have run out of space (even though bcachefs fs usage doesn't say so) and tried to do bcachefs device set-state rw /dev/nvme0n1p4, but that just hangs indefinitely without any kernel output.

I'll now try to reboot and see what happens, but wanted to paste the output here before it's all gone.

bcachefs fs usage

~~~ Filesystem: 677cf0a7-1abe-4ce3-876c-2ca63301229d Size: 5.89 TiB Used: 5.49 TiB Online reserved: 1.21 MiB Data type Required/total Durability Devices reserved: 1/0 [] 54.3 MiB btree: 1/3 6 [sdc1 sde1 sdh1] 1.69 GiB btree: 1/2 3 [sdc1 sdf1] 10.7 GiB btree: 1/2 3 [sdc1 nvme0n1p2] 18.5 GiB btree: 1/2 3 [sde1 sdb1] 3.91 GiB btree: 1/2 4 [sdc1 sde1] 5.09 GiB btree: 1/2 3 [sdc1 sdb1] 4.04 GiB btree: 1/2 3 [sde1 nvme0n1p2] 17.4 GiB btree: 1/2 3 [sde1 sdf1] 10.1 GiB user: 1/1 1 [nvme0n1p4] 70.7 GiB user: 1/1 2 [sde1] 86.3 GiB user: 1/1 1 [sdb1] 2.18 TiB user: 1/1 2 [sdc1] 92.4 GiB user: 1/1 2 [sdh1] 2.56 TiB user: 1/1 1 [nvme0n1p2] 411 GiB user: 1/1 1 [sdf1] 13.5 MiB cached: 1/1 2 [sdc1] 1.41 GiB cached: 1/1 1 [nvme0n1p2] 38.6 GiB cached: 1/1 2 [sdh1] 33.9 MiB cached: 1/1 1 [sdf1] 195 GiB cached: 1/1 2 [sde1] 1.37 GiB cached: 1/1 1 [nvme0n1p4] 220 GiB cached: 1/1 1 [sdb1] 2.29 GiB hdd.hdd1 (device 2): sdh1 ro data buckets fragmented free: 171 GiB 700503 sb: 3.00 MiB 13 252 KiB journal: 2.00 GiB 8192 btree: 577 MiB 2307 user: 2.56 TiB 10734302 423 MiB cached: 33.9 MiB 1032 parity: 0 B 0 stripe: 0 B 0 need_gc_gens: 0 B 0 need_discard: 0 B 0 capacity: 2.73 TiB 11446349 hdd.hdd2 (device 5): sdb1 rw data buckets fragmented free: 532 GiB 545108 sb: 3.00 MiB 4 1020 KiB journal: 8.00 GiB 8192 btree: 3.97 GiB 14765 10.4 GiB user: 2.18 TiB 2290881 1.06 GiB cached: 2.29 GiB 2637 parity: 0 B 0 stripe: 0 B 0 need_gc_gens: 0 B 0 need_discard: 0 B 0 capacity: 2.73 TiB 2861587 ssd.ssd1 (device 0): sdc1 rw data buckets fragmented free: 4.67 GiB 19133 sb: 3.00 MiB 13 252 KiB journal: 954 MiB 3815 btree: 19.7 GiB 80848 user: 92.4 GiB 378758 40.9 MiB cached: 1.41 GiB 5846 parity: 0 B 0 stripe: 0 B 0 need_gc_gens: 0 B 0 need_discard: 0 B 0 capacity: 119 GiB 488413 ssd.ssd2 (device 1): sde1 rw data buckets fragmented free: 4.38 GiB 17941 sb: 3.00 MiB 13 252 KiB journal: 894 MiB 3577 btree: 18.8 GiB 77080 user: 86.3 GiB 353612 40.2 MiB cached: 1.37 GiB 5665 parity: 0 B 0 stripe: 0 B 0 need_gc_gens: 0 B 0 need_discard: 256 KiB 1 capacity: 112 GiB 457889 ssd.ssd3 (device 3): nvme0n1p4 ro data buckets fragmented free: 59.0 GiB 120896 sb: 3.00 MiB 7 508 KiB journal: 2.76 GiB 5653 btree: 0 B 0 user: 70.7 GiB 144883 47.4 MiB cached: 220 GiB 452183 parity: 0 B 0 stripe: 0 B 0 need_gc_gens: 0 B 0 need_discard: 0 B 0 capacity: 353 GiB 723622 ssd.ssd4 (device 4): nvme0n1p2 rw data buckets fragmented free: 19.5 GiB 40025 sb: 3.00 MiB 7 508 KiB journal: 3.91 GiB 8000 btree: 18.0 GiB 54347 8.57 GiB user: 411 GiB 841710 235 MiB cached: 38.6 GiB 79908 parity: 0 B 0 stripe: 0 B 0 need_gc_gens: 0 B 0 need_discard: 1.50 MiB 3 capacity: 500 GiB 1024000 ssd.ssd5 (device 6): sdf1 rw data buckets fragmented free: 24.2 GiB 49619 sb: 3.00 MiB 7 508 KiB journal: 1.82 GiB 3726 btree: 10.4 GiB 24166 1.40 GiB user: 13.5 MiB 33 3.01 MiB cached: 195 GiB 399394 parity: 0 B 0 stripe: 0 B 0 need_gc_gens: 0 B 0 need_discard: 1.50 MiB 3 capacity: 233 GiB 476948 ~~~

[Lykos153]

Ok, so I'm back, the filesystem could be mounted read-write during boot and then I was indeed greeted with "filesystem full" messages. After setting all devices rw again, everything is back to normal. 🥳

Now I'll re-try the evacuation after adding a temp drive to the filesystem.

[Lykos153]

Ok so what I actually ended up doing (instead of adding a temp drive) was setting sdh1 to rw. After that, the evacuation went through without any issues. Though I don't know why it failed in first place because on /dev/sdb1 there were 532 GiB free which should have been more than enough to evacuate a 353 GiB drive.

[Lykos153]

Btw "and it's working well so far" was an understatement. I'm really impressed how robust this file system is. There were multiple points where I thought it'd now be FUBAR´d but actually it came back completely healthy every time. I'm really enthusiastic about bcachefs. Thank you for all your work!