Intermittent "target is busy" when creating btrfs subvolumes

[MarekPikula]

Occasionally, when creating btrfs subvolumes (a lot of them) the following error is displayed and nixos-anywhere deploy fails:

...
Create subvolume '/tmp/tmp.8pVv2w6CPt/var-cache'
++ umount /tmp/tmp.8pVv2w6CPt
++ rm -rf /tmp/tmp.8pVv2w6CPt
++ mktemp -d
+ MNTPOINT=/tmp/tmp.Z6H7oclcoI
+ mount /dev/bogucice-central-vg/root /tmp/tmp.Z6H7oclcoI -o subvol=/
+ trap 'umount $MNTPOINT; rm -rf $MNTPOINT' EXIT
+ btrfs subvolume create /tmp/tmp.Z6H7oclcoI//var-log
Create subvolume '/tmp/tmp.Z6H7oclcoI/var-log'
++ umount /tmp/tmp.Z6H7oclcoI
umount: /tmp/tmp.Z6H7oclcoI: target is busy.
+ rm -rf /tmp/tmp.eqcU0kNiK3

I guess it's because the previous subvolume creation is not finished at the point of umount, so maybe a sync before umount would do the job? Mind that I'm running it on a rather performant NVME, so it's not an issue of slow storage.

[MarekPikula]

Ah, I know what's the reason for slower-than-normal operation. I put the btrfs on a LVM volume with active dm-integrity, which, when created, recalculates checksums for the entire disk, which results in a slower IO.

[Lassulus]

ah interesting, I thought a umount always does a sync first. can you test if a sync before unmounting does indeed solve the issue?

[Lassulus]

otherwise a config to reproduce the issue inside a test would be helpful

[MarekPikula]

How can I add the sync? I'm quite new to Nix, so I would appreciate how I could do it.

Here's my current config:

{ lib, ... }:

{
  disko.devices = {
    disk = {
      nvme0 = {
        type = "disk";
        device = "/dev/nvme0n1";
        content = {
          type = "gpt";
          partitions = {
            ESP = {
              size = "512M";
              type = "EF00";
              content = {
                type = "filesystem";
                format = "vfat";
                mountpoint = "/boot";
                mountOptions = [ "defaults" ];
              };
            };
            luks = {
              size = "100%";
              content = {
                type = "luks";
                name = "crypt0";
                settings.allowDiscards = true;
                passwordFile = "/tmp/secret.key";
                content = {
                  type = "lvm_pv";
                  vg = "lvm-vg";
                };
              };
            };
          };
        };
      };
      nvme1 = {
        type = "disk";
        device = "/dev/nvme1n1";
        content = {
          type = "gpt";
          partitions = {
            ESP = {
              size = "512M";
              type = "EF00";
              content = {
                type = "filesystem";
                format = "vfat";
                mountpoint = "/boot1";
                mountOptions = [ "defaults" ];
              };
            };
            luks = {
              size = "100%";
              content = {
                type = "luks";
                name = "crypt1";
                settings.allowDiscards = true;
                passwordFile = "/tmp/secret.key";
                content = {
                  type = "lvm_pv";
                  vg = "lvm-vg";
                };
              };
            };
          };
        };
      };
    };
    lvm_vg = {
      lvm-vg = {
        type = "lvm_vg";
        lvs = {
          root = {
            size = "100G";
            lvm_type = "raid1";
            extraArgs = [ "--raidintegrity" "y" ];

            content = {
              type = "btrfs";
              extraArgs = [ "-f" ];
              subvolumes = {
                "/.snapshots" = {
                  mountpoint = "/.snapshots";
                  mountOptions = [ "compress=zstd" "noatime" ];
                };
                "/root" = {
                  mountpoint = "/";
                  mountOptions = [ "compress=zstd" "noatime" ];
                };
                "/home" = {
                  mountpoint = "/home";
                  mountOptions = [ "compress=zstd" "noatime" ];
                };
                "/nix" = {
                  mountpoint = "/nix";
                  mountOptions = [ "compress=zstd" "noatime" ];
                };
                "/var-cache" = {
                  mountpoint = "/var/cache";
                  mountOptions = [ "compress=zstd" "noatime" ];
                };
                "/var-log" = {
                  mountpoint = "/var/log";
                  mountOptions = [ "compress=zstd" "noatime" ];
                };
                "/var-tmp" = {
                  mountpoint = "/var/tmp";
                  mountOptions = [ "compress=zstd" "noatime" ];
                };
              };
            };
          };

          virt-homeassistant = {
            size = "100G";
            lvm_type = "raid1";
            extraArgs = [ "--raidintegrity" "y" ];
          };
          virt-opnsense = {
            size = "20G";
            lvm_type = "raid1";
            extraArgs = [ "--raidintegrity" "y" ];
          };
          virt-proxy = {
            size = "25G";
            lvm_type = "raid1";
            extraArgs = [ "--raidintegrity" "y" ];
          };
          virt-backup-gateway = {
            size = "25G";
            lvm_type = "raid1";
            extraArgs = [ "--raidintegrity" "y" ];
          };
        };
      };
    };
  };
}

[Lassulus]

not sure if the root subvolume is supported, we had an issue about that 2 days ago: https://github.com/nix-community/disko/issues/442

[MarekPikula]

not sure if the root subvolume is supported, we had an issue about that 2 days ago: https://github.com/nix-community/disko/issues/442

Well, for me it works (including LVM RAID with dm-verity – here I only needed to add a bunch of kernel modules to initramfs) 🙂

[iFreilicht]

This is an edge-case, but it would still be nice to get a repro for this and even fix it properly.