Block device mount in guest (ZFS zvol)

[pierregillet]

I would like to access a ZFS zvol (block device) from the VM but can't figure it out for the life of me. I saw references to block devices mounting in the documentation, and this comment that was posted:

https://astro.github.io/microvm.nix/faq.html#how-do-i-let-the-microvm-user-access-block-devices

As there are multiple solutions to allowing block device access (user extraGroup or udev rules), we cannot check disk permissions at build time. At runtime the VMM will error anyway.

I've added some documentation. The next release will be breaking anyway.

Originally posted by @astro in https://github.com/astro/microvm.nix/issues/222#issuecomment-2027637863

To mount it, I have tried :

1. Adding microvm user to disk group

     users.users.microvm.extraGroups = [ "disk" ];

2. Then:
   • mounting my zvol as a microvm volume (does not seem correct, as volumes are only for FS mount, right ?)

     {
     mountPoint = "/mnt/myvol";
     image = "/dev/zvol/mypool/myvol";
     }

   • as a last solution, passing extra args to qemu

     microvm = {
      qemu.extraArgs = [
      # "-drive file=/dev/zvol/mypool/myzvol,format=raw,if=virtio"  # first try
      # "-drive file=/dev/zvol/mypool/myzvol,format=raw"  # second try
      # "-blockdev node-name=q1,driver=raw,file.driver=host_device,file.filename=/dev/zvol/mypool/myzvol -device virtio-blk,drive=q1"   # third try
      # ];
     };

Would you have any pointers on how to proceed ?

(disclaimer: I am rather new to nixos, and qemu, may not have seen the obvious)

[astro]

The extra group and the former config look right. What's the error message you are getting?

[pierregillet]

When I use the volume as I described, without the size attribute, I get the following error:

error: The option `microvm.volumes."[definition 1-entry 3]".size' is used but not defined.

When I set a dummy volume size (10240 in this case), the VM is built and starts as expected, and I don't see any error. However the volume does not seem to be mounted as I don't see it in the output of lsblk nor mount (in the VM), nor at the specified path

[astro]

That's weird because microvm.volumes end up in fileSystems: https://github.com/astro/microvm.nix/blob/main/nixos-modules/microvm/mounts.nix#L96

[pierregillet]

I checked and don't see it in the filesystems.nix file from the store, readable from the guest

FYI I have not set the following ZFS dataset options, as I believe they do not apply to block devices:

-o xattr=sa -o acltype=posixacl

talked about here: https://github.com/astro/microvm.nix/issues/246#issuecomment-2146356771

[astro]

What is the filesystems.nix file? NixOS options don't end up in /nix/store directly.

[pierregillet]

Ah, my bad. I found that with nix repl on my host

nix-repl> :p nixosConfigurations.nixos.config.microvm.vms.myVM.flake.nixosConfigurations.nixos.options.fileSystems.declarations

which shows:

[ "/nix/store/qwcxqcxfmf6nifz3yqg6rkn2h6mf65x9-source/nixos/modules/tasks/filesystems.nix" "/nix/store/qwcxqcxfmf6nifz3yqg6rkn2h6mf65x9-source/nixos/modules/tasks/encrypted-devices.nix" "/nix/store/qwcxqcxfmf6nifz3yqg6rkn2h6mf65x9-source/nixos/modules/system/boot/stage-1.nix" ]

and I looked into /nix/store/qwcxqcxfmf6nifz3yqg6rkn2h6mf65x9-source/nixos/modules/tasks/filesystems.nix, but it seems that it's not related to microVM.

Sorry for the mixup.

[pierregillet]

Following the path in the ExecStart of the systemd service starting the VM (/var/lib/microvms/myVM/current/bin/microvm-run), I confirm that the volume does not appear in the qemu command as a -drive parameter. (I do however have the rootfs and the store overlay drives as expected).

I do see it appear in the volumes attribute set in the nix repl, as expected:

nix-repl> :p nixosConfigurations.myVM.config.microvm.volumes      
[ { autoCreate = true; fsType = "ext4"; image = "rootfs.img"; label = null; mountPoint = "/"; size = 10240; } 
  { autoCreate = true; fsType = "ext4"; image = "nix-store-overlay.img"; label = null; mountPoint = "/nix/.rw-store"; size = 10240; } 
  { autoCreate = true; fsType = "ext4"; image = "/dev/zvol/mypool/myzvol"; label = null; mountPoint = "/dev/zvol/mypool/myzvol"; size = 10240; } ]

I keep looking in that direction, thanks for the pointers :)

edit: I see that a fsType is added, when I expect the raw block device to be available in the VM, is it expected ? Is it really possible to have the raw block device usable from the guest "as is" without mounting it as a filesystem ?

[pierregillet]

I am confused, as I don't find the fileSystems attribute set you cited when using the repl:

nix-repl> :p nixosConfigurations.myVM.config.microvm.  (autocomplete command)
nixosConfigurations.myVM.config.microvm.balloonMem            nixosConfigurations.myVM.config.microvm.hypervisor            nixosConfigurations.myVM.config.microvm.socket
nixosConfigurations.myVM.config.microvm.bootDisk              nixosConfigurations.myVM.config.microvm.initrdPath            nixosConfigurations.myVM.config.microvm.storeDisk
nixosConfigurations.myVM.config.microvm.cloud-hypervisor      nixosConfigurations.myVM.config.microvm.interfaces            nixosConfigurations.myVM.config.microvm.storeDiskType
nixosConfigurations.myVM.config.microvm.cpu                   nixosConfigurations.myVM.config.microvm.kernel                nixosConfigurations.myVM.config.microvm.storeOnDisk
nixosConfigurations.myVM.config.microvm.crosvm                nixosConfigurations.myVM.config.microvm.kernelParams          nixosConfigurations.myVM.config.microvm.user
nixosConfigurations.myVM.config.microvm.declaredRunner        nixosConfigurations.myVM.config.microvm.mem                   nixosConfigurations.myVM.config.microvm.vcpu
nixosConfigurations.myVM.config.microvm.deploy                nixosConfigurations.myVM.config.microvm.optimize              nixosConfigurations.myVM.config.microvm.volumes
nixosConfigurations.myVM.config.microvm.devices               nixosConfigurations.myVM.config.microvm.preStart              nixosConfigurations.myVM.config.microvm.vsock
nixosConfigurations.myVM.config.microvm.forwardPorts          nixosConfigurations.myVM.config.microvm.prettyProcnames       nixosConfigurations.myVM.config.microvm.writableStoreOverlay
nixosConfigurations.myVM.config.microvm.graphics              nixosConfigurations.myVM.config.microvm.qemu
nixosConfigurations.myVM.config.microvm.guest                 nixosConfigurations.myVM.config.microvm.runner
nixosConfigurations.myVM.config.microvm.hugepageMem           nixosConfigurations.myVM.config.microvm.shares

Isn't it supposed to be here ?

Edit: I am building the microVM declaratively

[pierregillet]

I have made the error of rebuilding the declarative VM with sudo nixos-rebuild switch and restarting the service and forgetting to update explicitely the VM with sudo microvm -u myVM. Apologies for the stupid error, and thank you for your quick replies.

I am keeping the issue for now and will close it once I get it working, should be quick.

[pierregillet]

The microMV now outputs an error on startup, as it tries to mount my block device as a ext4 filesystem. I am trying to have the raw block device on the VM, not mounted as a filesystem.

Sep 12 01:25:42 nixos microvm@myVM[197643]:          Mounting /dev/zvol/myzpool/myzvol...
Sep 12 01:25:42 nixos microvm@myVM[197643]:          Starting Virtual Console Setup[    4.992321] EXT4-fs (vdc): VFS:     Can't find ext4 filesystem
Sep 12 01:25:42 nixos microvm@myVM[197643]: ...
Sep 12 01:25:42 nixos microvm@myVM[197643]: [FAILED] Failed to mount /dev/zvol/zpool/myzvol.
Sep 12 01:25:42 nixos microvm@myVM[197643]: See 'systemctl status "dev-zvol-zpool-myzvol.mount"' for details.
Sep 12 01:25:42 nixos microvm@myVM[197643]: [DEPEND] Dependency failed for Local File Systems.
[...]
Sep 12 01:25:42 nixos microvm@myVM[197643]: You are in emergency mode. After logging in, type "journalctl -xb" to view
Sep 12 01:25:42 nixos microvm@myVM[197643]: system logs, "systemctl reboot" to reboot, or "exit"
Sep 12 01:25:42 nixos microvm@myVM[197643]: to continue bootup.
Sep 12 01:25:42 nixos microvm@myVM[197643]: Give root password for maintenance

How can I mount it as a raw block device in the guest ?

[astro]

I miss the nixos-modules/microvm/mounts.nix module in your nixosConfigurations.nixos.config.microvm.vms.myVM.flake.nixosConfigurations.nixos.options.fileSystems.declarations output. Are you sure you imported it?

The /dev/zvol/... path is not supposed to end up in the VM's fstab. Instead, the dev nodes are going to be /dev/vd[a-z]

[pierregillet]

You are right that nixos-modules/microvm/mounts.nix is not in my nixosConfigurations.nixos.config.microvm.vms.myVM.flake.nixosConfigurations.nixos.options.fileSystems.declarations output:

nix-repl> :p nixosConfigurations.nixos.config.microvm.vms.myVM.flake.nixosConfigurations.nixos.options.fileSystems.declarations
[ "/nix/store/qwcxqcxfmf6nifz3yqg6rkn2h6mf65x9-source/nixos/modules/tasks/filesystems.nix" 
  "/nix/store/qwcxqcxfmf6nifz3yqg6rkn2h6mf65x9-source/nixos/modules/tasks/encrypted-devices.nix" 
  "/nix/store/qwcxqcxfmf6nifz3yqg6rkn2h6mf65x9-source/nixos/modules/system/boot/stage-1.nix" ]

However I do see the dev node appearing in the guest fileSystems declaration (I guess that's coherent with it ending up in the fstab):

nix-repl> :p nixosConfigurations.nixos.config.microvm.vms.myVM.flake.nixosConfigurations.myVM.options.fileSystems.definitions
[...]
"/" = { device = "/dev/vda"; fsType = "ext4";}; 
"/dev/zvol/mypool/myvol" = { device = "/dev/vdc"; fsType = "ext4";};
"/nix/.rw-store" = { device = "/dev/vdb"; fsType = "ext4"; neededForBoot = true;};
[...]

I miss the nixos-modules/microvm/mounts.nix module [...] Are you sure you imported it?

If it has to be done explicitly I didn't import it. I poked around and tried to find, but I can't find how to import this module ?

(Adding microvm.nixosModules.microvm.mounts to my host's modules list gets me an error)

[astro]

It's just microvm.nixosModules.microvm that imports that mounts.nix file. Did you import that?

[pierregillet]

Yes I did import that in my guest config. I feel like it would be easier at this point that I share the config I use:

This is, roughly, the config I use for the host:

nixos = nixpkgs.lib.nixosSystem {
  inherit pkgs;
  specialArgs = { inherit inputs; };
  modules = [
    ./configuration.nix
    microvm.nixosModules.host
    {
      systemd.network.enable = true;
      systemd.network.networks."10-lan" = {
        matchConfig.Name = ["<interface>" "myVM*"];
        networkConfig = {
          Bridge = "br0";
        };
      };

      systemd.network.netdevs."br0" = {
        netdevConfig = {
          Name = "br0";
          Kind = "bridge";
        };
      };

      systemd.network.networks."10-lan-bridge" = {
        matchConfig.Name = "br0";
        networkConfig = {
          Address = ["<IP>"];
          Gateway = "<GW>";
          DNS = ["<DNS>"];
          IPv6AcceptRA = true;
        };
        linkConfig.RequiredForOnline = "routable";
      };
      microvm.vms = {
        myVM = {
          # Host build-time reference to where the MicroVM NixOS is defined
          # under nixosConfigurations
          flake = self;
          # # Specify from where to let `microvm -u` update later on
          updateFlake = "git+file:///etc/nixos";
        };
      };
      # Allow access to ZFS block volume
      users.users.microvm.extraGroups = [ "disk" ];
    }
  ];
};

And this is, roughly, the config I use for the guest:

myVM = nixpkgs.lib.nixosSystem {
  system = "x86_64-linux";
  modules = [
    microvm.nixosModules.microvm
    ({ config, pkgs, ... }: {
      systemd.services.initialConfig = {
        description = "Copy configuration into microvm";
        wantedBy = [ "multi-user.target" ];
        unitConfig.ConditionPathExists = "!/root/flake.nix";
        serviceConfig.Type = "oneshot";
        script = ''
          mkdir -p /etc/nixos
          cp --no-preserve=mode ${./flake.nix} /etc/nixos/flake.nix
          cp --no-preserve=mode ${./flake.lock} /etc/nixos/flake.lock
          cp --no-preserve=mode ${./configuration-myVM.nix} /etc/nixos/configuration.nix
        '';
      };
      networking.hostName = "myVM";
      users.users.root.password = "";
      nix = {
        nixPath = [ "nixpkgs=${nixpkgs}" ];
        extraOptions = ''
          experimental-features = nix-command flakes
          accept-flake-config = true
        '';
      };
      environment.systemPackages = with pkgs; [ magic-wormhole bore-cli ];
      services.logind.extraConfig = "RuntimeDirectorySize=2G";
      services.getty.autologinUser = "root";
      systemd.network.enable = true;
      systemd.network.networks."20-lan" = {
        matchConfig.Type = "ether";
        networkConfig = {
          Address = ["<IP>"];
          Gateway = "<GW>";
          DNS = ["<DNS>"];
          IPv6AcceptRA = true;
          DHCP = "no";
        };
      };
      microvm = {
        vcpu = 4;
        interfaces = [
          {
            type = "tap";
            id = "myVM";
            mac = "<MAC>";
          }
        ];
        balloonMem = 4096;
        volumes = [
          {
            mountPoint = "/";
            image = "rootfs.img";
            size = 10240;
          }
          {
            image = "nix-store-overlay.img";
            mountPoint = config.microvm.writableStoreOverlay;
            size = 10240;
          }
          {
            mountPoint = "/dev/zvol/mypool/myzvol";
            image = "/dev/zvol/mypool/myzvol";
            size = 10240;
          }
        ];
      };
    })
  ];
}

Everything seems to be working as expected, except this last block device mount that is mounted as a filesystem on the guest

[astro]

mountPoint should not be the same as image but a directory's path inside the VM.

BTW, what are you trying to achieve by copying the Nix configuration into the VM? The VM is supposed to be built by the host.

[pierregillet]

what are you trying to achieve by copying the Nix configuration into the VM?

It was not supposed to be there, I removed it (it was from one of my first tests). Sorry for the mess!

mountPoint should not be the same as image but a directory's path inside the VM.

Understood, I switched back to /mnt (existing, empty directory) as mountpoint, with the same error (guest has this volume in its fstab)

[astro]

Can please post the mount error message?

[pierregillet]

With the command sudo journalctl -xeb on the host:

Sep 15 22:38:15 nixos microvm@k0s-node0[400295]: [  OK  ] Finished File System Check on /dev/vdc.
Sep 15 22:38:15 nixos microvm@k0s-node0[400295]:          Mounting /mnt...
Sep 15 22:38:15 nixos microvm@k0s-node0[400295]:          Starting Virtual Console Setup...
Sep 15 22:38:15 nixos microvm@k0s-node0[400295]: [    5.146738] EXT4-fs (vdc): VFS: Can't find ext4 filesystem
Sep 15 22:38:15 nixos microvm@k0s-node0[400295]: [FAILED] Failed to mount /mnt.
Sep 15 22:38:15 nixos microvm@k0s-node0[400295]: See 'systemctl status mnt.mount' for details.
Sep 15 22:38:15 nixos microvm@k0s-node0[400295]: [DEPEND] Dependency failed for Local File Systems.
[...]
Sep 15 22:38:15 nixos microvm@k0s-node0[400295]: You are in emergency mode. After logging in, type "journalctl -xb" to view
Sep 15 22:38:15 nixos microvm@k0s-node0[400295]: system logs, "systemctl reboot" to reboot, or "exit"
Sep 15 22:38:15 nixos microvm@k0s-node0[400295]: to continue bootup.
Sep 15 22:38:15 nixos microvm@k0s-node0[400295]: Give root password for maintenance

And since the VM is in emergency mode, I can't connect via SSH and get the full error message (if you have a tip there to get a shell in the VM I'd be interested) (Edit: seems like dropbear would allow SSH in emergency mode, I might look into setting that up if having a shell in the VM would be helpful)

[astro]

microvm.nix does not detect if it's a zvol, therefore you must create it manually and also run mkfs.ext4 yourself.

PR welcome :-)

[pierregillet]

I was, and am, considering contributing however I can, despite my very modest level here ;-) still very much in the learning process. In any case, thanks for all the help, much appreciated! :-)

microvm.nix does not detect if it's a zvol, therefore you must create it manually and also run mkfs.ext4 yourself.

Ah! to be sure we are on the same page, for such a PR, microvm would have to see that it is a zvol, but both cases would be possible:

• mounting the filesystem contained in the zvol
• mounting the zvol itself (not its filesystem) to have access to the raw zvol (block device) in the VM

The first is what it tries to do right now, but the second would require an additional parameter in the volumes attribute set, right ? Something like a boolean microvm.volumes.*.rawBlockDevice ?

[astro]

Not necessary, microvm.volumes.*.mountPoint can be left null if it shall not be mounted.

[pierregillet]

Ah, makes sense, thanks. I'll try to look into it.