Wayland VMs / wayland-proxy-virtwl virtio-gpu proxy support

[0xc1c4da]

microvm.nix could offer a clean way to offer reasonably secure desktop computing experience to existing Nix installations. Through the support Wayland GUI VM compartmentalization on personal computers, similar to Qubes OS.

Approaches have leveraged Nix to implement similar features, ie Spectrum OS and qubes-lite using Wayland. The main issues with these approaches is that Spectrum requires installing Spectrum from scratch, is under development and is difficult to retrofit onto an existing Nix installation, and qubes-lite is a personal script hacked together.

Rather than #57 where the gpu is passed through to a vm, the gpu can be shared across vms, and the vm's Wayland clients can be proxied to the host's compositor, using wayland-proxy-virtwl --virtio-gpu.

crosvm supports virtio-gpu natively, and Spectrum has patches that enables it for cloud-hypervisor. Wayland-proxy-virtwl is already included in nixpkgs pkgs/tools/wayland/wayland-proxy-virtwl/default.nix.

Some examples of its usage:

• https://gitlab.com/talex5/qubes-lite/-/blob/master/default.nix#L183
• https://spectrum-os.org/git/spectrum/tree/release/checks/wayland/default.nix?h=wayland&id=aefa4eb3d4618a1e6d8e4d2677fb1a97d3b80f58#n18

[astro]

I have pushed some basic infrastructure work to the graphics branch. If you say this is useful already, I can merge it.

In general the virtio-gpu Wayland situation seems a bit outdated and thus flaky. The patches that improved things do no longer apply to the latest crosvm/cloud-hypervisor versions. Let us keep track of the situation...

[alyssais]

The patches that improved things do no longer apply to the latest crosvm/cloud-hypervisor versions.

crosvm shouldn't need any patches, but there is a serious unresolved bug with crosvm as the VMM.

I've just updated the Spectrum Cloud Hypervisor patches to be compatible with CH 30 and 31, but there are some important caveats to using those patches:

• I can't commit to supporting any particular CH/crosvm versions at any particular time.
• They're not done yet (although I'm in the final stretch now — probably a matter of weeks before my todo list is cleared). In particular, I think they still don't work at all on ARM.

[0xc1c4da]

Wow thanks! Probably not ready for merge, unsure if user error, but I haven't been able to successfully proxy a client to the host.

in my /etc/nixos/flake.nix I have

nixosConfigurations.vm-browser = nixpkgs.lib.nixosSystem {
        inherit system;
        modules = [
          ./vms/vm-browser.nix
          microvm.nixosModules.microvm
          {
            networking.hostName = "vm-browser";
            users.users.root.password = "";
            microvm = {
              hypervisor = "crosvm";
              graphics.enable = true;
              vcpu = 2;
              mem = 2048;
              balloonMem = 4096;

              volumes = [ {
                mountPoint = "/var";
                image = "var.img";
                size = 256;
              } ];

              shares = [ 
                {
                  proto = "virtiofs";
                  tag = "ro-store";
                  source = "/nix/store";
                  mountPoint = "/nix/.ro-store";
                } 
              ];
            };
          }
        ];
      };

where ./vms/vm-browser.nix contains:

{ config, pkgs, stdenv, ... }:

{
    environment.systemPackages = with pkgs; [
        hello-wayland
    ];
}

then running $ doas microvm -r vm-browser resulted in booting with this appearing in logs

...
crosvm[24784]: failed to connect to display
[2023-04-11T10:37:40.735889655+02:00 ERROR devices::virtio::gpu] failed to open display: failed to connect to compositor
[2023-04-11T10:37:40.735905809+02:00 ERROR devices::virtio::gpu] failed to open display: unsupported by the implementation
...

I tried running this under SDDM/KDE Plasma, without further information, I swapped out for GDM/Gnome and same issue.

in the vm, run-wayland-proxy of course wouldn't work without display

$ run-wayland-proxy hello-wayland
[  133.795936] [drm:virtio_gpu_dequeue_ctrl_func [virtio_gpu]] *ERROR* response 0x1200 (command 0x207)
[  133.800419] [drm:virtio_gpu_dequeue_ctrl_func [virtio_gpu]] *ERROR* response 0x1200 (command 0x207)
[  133.801201] [drm:virtio_gpu_dequeue_ctrl_func [virtio_gpu]] *ERROR* response 0x1200 (command 0x207)
[  133.801835] [drm:virtio_gpu_dequeue_ctrl_func [virtio_gpu]] *ERROR* response 0x1200 (command 0x207)

I also noticed when shutting down the vm [2023-04-11T11:16:29.552107165+02:00 WARN base::sys::unix::net] failed to remove control socket file: Os { code: 13, kind: PermissionDenied, message: "Permission denied" }

I also tried doing the above but manually running the cross vm gpu process (afterrm vm-browser-gpu.sock) /var/lib/microvms/vm-browser]$ doas crosvm device gpu --socket vm-browser-gpu.sock --wayland-sock $XDG_RUNTIME_DIR/$WAYLAND_DISPLAY --params '{"context-types":"virgl:virgl2:cross-domain"}'

I see no chatter from this process

[0xc1c4da]

Ah nevermind, I just realised the code is relevant to declared runners, running with nix run, works!

[astro]

I have added an example: nix run github:astro/microvm.nix/graphics#graphics

In the long run, more useful would a wrapper like nix run ...#run-graphics nixpkgs#package

[0xc1c4da]

I setup the following in a vm that allows for QT/GTK/Electron applications to run out of the box and 'normally' without run-wayland-proxy

    environment.sessionVariables = {
        WAYLAND_DISPLAY = "wayland-1";
        # DISPLAY = ":0";
        QT_QPA_PLATFORM = "wayland"; # Qt Applications
        GDK_BACKEND = "wayland"; # GTK Applications
        XDG_SESSION_TYPE = "wayland"; # Electron Applications
        SDL_VIDEODRIVER = "wayland";
        CLUTTER_BACKEND = "wayland";

    };

    systemd.user.services.wayland-proxy = {
        enable = true;
        description = "Wayland Proxy";
        serviceConfig = with pkgs; {
            # Environment = "WAYLAND_DISPLAY=wayland-1";
            ExecStart = "${wayland-proxy-virtwl}/bin/wayland-proxy-virtwl --virtio-gpu"; #  --x-display=0
        };
        wantedBy = [ "default.target" ];
    };

    environment.systemPackages = with pkgs; [
        xdg-utils # Required

        librewolf-wayland
        keepassxc
        foot
        konsole
        obsidian # requires --ozone-platform-hint=auto uses XDG_SESSION_TYPE or setting ~/.config/electron-flags.conf
    ];

I haven't been able to successfully get EGL working (or if it's even supported).

There's still some work to get some QT/GTK functionality, but perhaps that's best left to the user, where I think the service and the environment variables are probably worth including, wdyt?

Currently on the host, there is a crosvm gpu window popping up for every vm, I can manually hide them, do we know if it's possible to share this process (as a service) ?

[astro]

The bit of JSON we pass to crosvm device gpu contains the same fields that can be passed to crosvm when running the GPU stuff in-process. I found them to be kinda ineffective, however. :-( Happy tinkering!

  --gpu             (EXPERIMENTAL) Comma separated key=value pairs for setting
                    up a virtio-gpu device
                    Possible key values:
                        backend=(2d|virglrenderer|gfxstream) - Which backend to
                           use for virtio-gpu (determining rendering protocol)
                        displays=[[GpuDisplayParameters]] - The list of virtual
                            displays to create. See the possible key values for
                            GpuDisplayParameters in the section below.
                        context-types=LIST - The list of supported context
                          types, separated by ':' (default: no contexts enabled)
                        width=INT - The width of the virtual display connected
                           to the virtio-gpu.
                           Deprecated - use `displays` instead.
                        height=INT - The height of the virtual display
                           connected to the virtio-gpu.
                           Deprecated - use `displays` instead.
                        egl[=true|=false] - If the backend should use a EGL
                           context for rendering.
                        glx[=true|=false] - If the backend should use a GLX
                           context for rendering.
                        surfaceless[=true|=false] - If the backend should use a
                            surfaceless context for rendering.
                        angle[=true|=false] - If the gfxstream backend should
                           use ANGLE (OpenGL on Vulkan) as its native OpenGL
                           driver.
                        vulkan[=true|=false] - If the backend should support
                           vulkan
                        wsi=vk - If the gfxstream backend should use the Vulkan
                           swapchain to draw on a window
                        cache-path=PATH - The path to the virtio-gpu device
                           shader cache.
                        cache-size=SIZE - The maximum size of the shader cache.
                        pci-bar-size=SIZE - The size for the PCI BAR in bytes
                           (default 8gb).
                    Possible key values for GpuDisplayParameters:
                        mode=(borderless_full_screen|windowed[width,height]) -
                           Whether to show the window on the host in full
                           screen or windowed mode. If not specified, windowed
                           mode is used by default. "windowed" can also be
                           specified explicitly to use a window size different
                           from the default one.
                        hidden[=true|=false] - If the display window is
                           initially hidden (default: false).
                        refresh-rate=INT - Force a specific vsync generation
                           rate in hertz on the guest (default: 60)
                        dpi=[INT,INT] - The horizontal and vertical DPI of the
                           display (default: [320,320])
                        horizontal-dpi=INT - The horizontal DPI of the display
                           (default: 320)
                           Deprecated - use `dpi` instead.
                        vertical-dpi=INT - The vertical DPI of the display
                           (default: 320)
                           Deprecated - use `dpi` instead.

[astro]

While far from being perfect and sustainable, this feature is that much nice to have that I polished it a bit and merged it into main.