search

koverstreet / bcachefs-tools

http://bcachefs.org
GNU General Public License v2.0
116 stars 88 forks source link

bcachefs device add yields inconsistent FS UUID in udev db and disk #267

Open tasleson opened 3 months ago

tasleson commented 3 months ago

Note: using bcachefs-tools 3ac510f6a41feb1b695381fa30869d557c00b822 with fedora 39 6.8.9-200 kernel

If you format a 2 device FS, eg.

# ./bcachefs format /dev/mapper/mpathc /dev/mapper/mpathd --replicas=2

format out shows FS UUID = 90dac8ea-7960-4cb6-b92e-3f86b7eed350

we check using blkid -p and they match our expectations

# blkid -p /dev/mapper/mpathc
/dev/mapper/mpathc: UUID="90dac8ea-7960-4cb6-b92e-3f86b7eed350" VERSION="1.4" BLOCK_SIZE="4096" FSBLOCKSIZE="4096" UUID_SUB="aaa03f60-0900-49ec-b683-1053b67f01cc" FSSIZE="2000409329664" TYPE="bcachefs" USAGE="filesystem"
# blkid -p /dev/mapper/mpathd
/dev/mapper/mpathd: UUID="90dac8ea-7960-4cb6-b92e-3f86b7eed350" VERSION="1.4" BLOCK_SIZE="4096" FSBLOCKSIZE="4096" UUID_SUB="7d84a5a6-1118-496f-9350-199cee694c71" FSSIZE="2000409329664" TYPE="bcachefs" USAGE="filesystem"

mount it:

# ./bcachefs mount  /dev/mapper/mpathc /mnt/bcachefs

add another device to it

./bcachefs device add /mnt/bcachefs/ /dev/mapper/mpathb

checks it's blkid info:

# blkid -p /dev/mapper/mpathb
/dev/mapper/mpathb: UUID="e165c540-ea4a-45d6-8d7d-c412d0202840" VERSION="1.4" BLOCK_SIZE="4096" FSBLOCKSIZE="4096" UUID_SUB="020478e0-5a12-4a60-9c6c-477944787107" FSSIZE="1000204664832" TYPE="bcachefs" USAGE="filesystem"

and find that its FS UUID is different!

The udev db for this block device is also showing ID_FS_UUID=e165c540-ea4a-45d6-8d7d-c412d0202840

If we un-mount the bcachefs FS and check FS UUID on disk we find

# umount /mnt/bcachefs 
# blkid -p /dev/mapper/mpathb
/dev/mapper/mpathb: UUID="90dac8ea-7960-4cb6-b92e-3f86b7eed350" VERSION="1.4" BLOCK_SIZE="4096" FSBLOCKSIZE="4096" UUID_SUB="020478e0-5a12-4a60-9c6c-477944787107" FSSIZE="3000613994496" TYPE="bcachefs" USAGE="filesystem"

that the FS UUID now matches the original formatted FS.

However, the udev db still shows ID_FS_UUID=e165c540-ea4a-45d6-8d7d-c412d0202840

So if we now try to mount the FS using 1 of the device nodes or the FS UUID we fail to find all of the devices!

 ./bcachefs mount -v /dev/mapper/mpathc /mnt/bcachefs/
DEBUG - bcachefs::commands::mount: Walking udev db!
DEBUG - bcachefs::commands::mount: enumerating devices with UUID 90dac8ea-7960-4cb6-b92e-3f86b7eed350
INFO - bcachefs::commands::mount: mounting with params: device: /dev/dm-10:/dev/dm-17, target: /mnt/bcachefs/, options: 
DEBUG - bcachefs::commands::mount: parsing mount options: 
INFO - bcachefs::commands::mount: mounting bcachefs filesystem, /mnt/bcachefs/
INFO - bcachefs::commands::mount: mounting filesystem
ERROR - bcachefs::commands::mount: Fatal error: Invalid argument

dmesg shows:

[11617.259122] bcachefs: bch2_fs_open() bch_fs_open err opening /dev/dm-10: insufficient_devices_to_start

We can get it to mount if we supply all the block devices, eg. ./bcachefs mount -v /dev/mapper/mpathc:/dev/mapper/mpathb:/dev/mapper/mpathd /mnt/bcachefs/

So the question is, why does a newly added device have a different FS UUID when added?

I'm not sure how to fix this, so that when the super block gets written out is has the correct FS UUID during device add.

tasleson commented 3 months ago

If we update the udev db, # udevadm control --reload-rules && udevadm trigger we can mount as expected

./bcachefs mount -v /dev/mapper/mpathc /mnt/bcachefs/
DEBUG - bcachefs::commands::mount: Walking udev db!
DEBUG - bcachefs::commands::mount: enumerating devices with UUID 90dac8ea-7960-4cb6-b92e-3f86b7eed350
INFO - bcachefs::commands::mount: mounting with params: device: /dev/dm-10:/dev/dm-17:/dev/dm-23, target: /mnt/bcachefs/, options: 
DEBUG - bcachefs::commands::mount: parsing mount options: 
INFO - bcachefs::commands::mount: mounting bcachefs filesystem, /mnt/bcachefs/
INFO - bcachefs::commands::mount: mounting filesystem
INFO - bcachefs::commands::mount: Successfully mounted