search

canonical / lxd

Powerful system container and virtual machine manager
https://canonical.com/lxd
GNU Affero General Public License v3.0
4.27k stars 912 forks source link

Inconsistencies on `/dev/disk/by-id/*` when using similar long device names and multipath #13701

Open hamistao opened 6 days ago

hamistao commented 6 days ago

Required information

Issue description

When using two block devices with names that share a prefix of 16 characters (after escaping) and also using multipath, the /dev/disk/by-id/* symlinks to those devices are identical resulting in a single symlink being created. The second device would now be accessed through /dev/dm-0 and not /dev/sdc. Although the latter exists, making a filesystem on it fails and returns /dev/sdc is apparently in use by the system; will not make a filesystem here!. That problem only occurs when multipath-tools is installed and removing it and restarting the VM fixes this. Note that the second device is still usable, but through /dev/dm-0. So this may not be critical but it should be worth some discussion to better understand what is happening.

Steps to reproduce

  1. lxc launch ubuntu:n vm --vm # Ubuntu images come with multipath-tools installed
  2. longName="long-device-name"
  3. lxc storage volume create default vol1 --type=block
  4. lxc storage volume create default vol2 --type=block
  5. sleep 30
  6. lxc exec vm -- systemctl is-system-running --wait
  7. lxc config device add vm "${longName}1" disk pool=default source=vol1
  8. lxc config device add vm "${longName}2" disk pool=default source=vol2
  9. lxc exec vm -- mkfs.ext4 /dev/sdc # Fails
  10. lxc exec vm -- apt autopurge multipath-tools
  11. lxc restart vm
  12. lxc exec vm -- mkfs.ext4 /dev/sdc # Succeeds
hamistao commented 6 days ago

Please correct me if I am wrong @simondeziel. This seems to be happening because when using a long block device name, the /dev/disk/by-id only includes the first 16 characters of the device name (after escaping). For example, using long-device-name1 as block device name would result in its path on /dev/disk/by-id to look like 0QEMU_QEMU_HARDDISK_lxd_long--device--na, and the same would happen for a second added device named long-device-name2, overwriting the first /dev/disk/by-id link and leaving the first device without a link on /dev/disk/by-id. This could be making multipath-tools assume both devices are the same and creating the inconsistencies described above.

tomponline commented 6 days ago

@hamistao I dont think there is much we can do about this I'm afraid.

hamistao commented 6 days ago

@tomponline I agree, I don't think we can't fix this without changing /dev/disk/by-id paths. I wanted to open this in case someone else could think of a solution.

tomponline commented 6 days ago

If this is a limitation of udev there isn't much we can do.

simondeziel commented 6 days ago

@hamistao as you described, I think that due the /dev/disk/by-id symlinks ending up as the same, the 2nd disk sharing the name prefix is the one taking over the by-id symlink. This in turn seems to hint multipath into thinking there are many paths leading to the same disk.