blockdev: ioctl error on BLKRRPART: Device or resource busy

[jkirk]

I try to run grml-debootstrap on a Proxmox VM and get the following error:

root@grml ~ # grml-debootstrap -r bookworm -t /dev/sda1 --grub /dev/sda --hostname template-debian-bookworm --mirror http://mirror.hetzner.de/debian/packages
 * EFI support detected, but system seems to be running in BIOS mode.
 * grml-debootstrap [0.108] - Please recheck configuration before execution:

   Target:          /dev/sda1
   Install grub:    /dev/sda
   Install efi:     no
   Using release:   bookworm
   Using hostname:  template-debian-bookworm
   Using mirror:    http://mirror.hetzner.de/debian/packages
   Using arch:      amd64
   Config files:    /etc/debootstrap

   Important! Continuing will delete all data from /dev/sda1!

 * Is this ok for you? [y/N] y
 * Running mkfs.ext4  on /dev/sda1
mke2fs 1.47.0 (5-Feb-2023)
/dev/sda1 contains a ext4 file system
        last mounted on / on Tue Feb  6 10:14:45 2024
Proceed anyway? (y,N) y
Discarding device blocks: done                            
Creating filesystem with 4194048 4k blocks and 1048576 inodes
Filesystem UUID: 6f358ba8-5352-4b87-b0a8-7117e126097c
Superblock backups stored on blocks: 
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
        4096000

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done   

blockdev: ioctl error on BLKRRPART: Device or resource busy
Unexpected non-zero exit code 1 in /sbin/grml-debootstrap /sbin/grml-debootstrap /sbin/grml-debootstrap at line 1376 2132 0 detected!
last bash command: blockdev --rereadpt "$main_device"

root@grml ~ # mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=986344k,nr_inodes=246586,mode=755,inode64)
devpts on /dev/pts type devpts (rw,relatime,mode=622,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,size=202408k,mode=755,inode64)
/dev/sr0 on /run/live/medium type iso9660 (ro,noatime,nojoliet,check=s,map=n,blocksize=2048,iocharset=utf8)
/dev/loop0 on /run/live/rootfs/grml64-full.squashfs type squashfs (ro,noatime,errors=continue)
tmpfs on /run/live/overlay type tmpfs (rw,noatime,size=1012036k,mode=755,inode64)
overlay on / type overlay (rw,noatime,lowerdir=/run/live/rootfs/grml64-full.squashfs/,upperdir=/run/live/overlay/rw,workdir=/run/live/overlay/work,uuid=on)
tmpfs on /usr/lib/live/mount type tmpfs (rw,nosuid,nodev,noexec,relatime,size=202408k,mode=755,inode64)
/dev/sr0 on /usr/lib/live/mount/medium type iso9660 (ro,noatime,nojoliet,check=s,map=n,blocksize=2048,iocharset=utf8)
/dev/loop0 on /usr/lib/live/mount/rootfs/grml64-full.squashfs type squashfs (ro,noatime,errors=continue)
tmpfs on /usr/lib/live/mount/overlay type tmpfs (rw,noatime,size=1012036k,mode=755,inode64)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k,inode64)
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=32,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=3224)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,nosuid,nodev,relatime,pagesize=2M)
tracefs on /sys/kernel/tracing type tracefs (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
fusectl on /sys/fs/fuse/connections type fusectl (rw,nosuid,nodev,noexec,relatime)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,relatime,inode64)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=202404k,nr_inodes=50601,mode=700,inode64)

As a workaround I added sync + sleep 2 before blockdev --rereadpt "$main_device" https://github.com/grml/grml-debootstrap/blob/fa44c61d9c8b7a1559d409a0d6f8fbdcb7350cac/grml-debootstrap#L1376

I am not sure that this is the right way to fix this problem. In fact I have no idea, why this happens in the first place. Any idea how to fix this?

[jkirk]

Seems like udev kicks in:

root@grml ~ # while ! lsof +c0 /dev/sda; do sleep 0.1; done                                                                                                                                                     
COMMAND        PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
(udev-worker) 8271 root   18rR  BLK    8,0      0t0  215 /dev/sda
(udev-worker) 8271 root   19rR  BLK    8,0      0t0  215 /dev/sda

root@grml ~ # udevadm monitor
monitor will print the received events for:
UDEV - the event which udev sends out after rule processing
KERNEL - the kernel uevent

KERNEL[8081.262680] change   /devices/pci0000:00/0000:00:05.0/0000:01:01.0/virtio2/host0/target0:0:0/0:0:0:0/block/sda/sda1 (block)
UDEV  [8081.514711] change   /devices/pci0000:00/0000:00:05.0/0000:01:01.0/virtio2/host0/target0:0:0/0:0:0:0/block/sda/sda1 (block)
KERNEL[8081.537754] remove   /devices/pci0000:00/0000:00:05.0/0000:01:01.0/virtio2/host0/target0:0:0/0:0:0:0/block/sda/sda1 (block)
KERNEL[8081.538049] change   /devices/pci0000:00/0000:00:05.0/0000:01:01.0/virtio2/host0/target0:0:0/0:0:0:0/block/sda (block)
KERNEL[8081.538269] add      /devices/pci0000:00/0000:00:05.0/0000:01:01.0/virtio2/host0/target0:0:0/0:0:0:0/block/sda/sda1 (block)
UDEV  [8081.746240] remove   /devices/pci0000:00/0000:00:05.0/0000:01:01.0/virtio2/host0/target0:0:0/0:0:0:0/block/sda/sda1 (block)
UDEV  [8081.799125] change   /devices/pci0000:00/0000:00:05.0/0000:01:01.0/virtio2/host0/target0:0:0/0:0:0:0/block/sda (block)
UDEV  [8082.042753] add      /devices/pci0000:00/0000:00:05.0/0000:01:01.0/virtio2/host0/target0:0:0/0:0:0:0/block/sda/sda1 (block)

Instead of sync and sleep 1, I tried udevadm settle which seems to do the trick. Before creating a PR for this specific problem, I wonder where else do we need this?

[zeha]

Instead of sync and sleep 1, I tried udevadm settle which seems to do the trick.

I think this needs a retry instead, because lots of things try to access new devices...

[jkirk]

Not sure, because an instant retry failed too:

root@grml ~ # grml-debootstrap -r bookworm -t /dev/sda1 --grub /dev/sda --hostname template-debian-bookworm --mirror http://mirror.hetzner.de/debian/packages; blockdev --rereadpt "$main_device"
[...]
blockdev: ioctl error on BLKRRPART: Device or resource busy
Unexpected non-zero exit code 1 in /sbin/grml-debootstrap /sbin/grml-debootstrap /sbin/grml-debootstrap at line 1376 2132 0 detected!
last bash command: blockdev --rereadpt "$main_device"
blockdev: ioctl error on BLKRRPART: Device or resource busy
root@grml ~ # 

FTR, the problem did not occur when I stopped the udev event handler before and starting it after grml-debootstrap:

root@grml ~ # udevadm control -s
root@grml ~ # grml-debootstrap -r bookworm -t /dev/sda1 --grub /dev/sda --hostname template-debian-bookworm --mirror http://mirror.hetzner.de/debian/packages;
[...]
   * Finished execution of grml-debootstrap. Enjoy your Debian system.
root@grml ~ # udevadm control -S

[zeha]

instant retry

Retry of the entire grml-debootstrap call or of blockdev with a sleep loop?