Cannot start a VM with manually built multipass

[nikelborm]

Hi, I just tried to install multipass on my arch linux machine without snap. I used the version from AUR. I successfully built it and everything seemed fine until I tried to actually launch a VM. I made sure the daemon started via

sudo systemctl status multipassd

When I try to start a vm like this:

multipass launch -n foo2

It throws following error

launch failed: The following errors occurred:
qemu: could not load PC BIOS 'OVMF.fd'
foo2: shutdown called while starting

After some reading in the internet i found that maybe something wrong with my /usr/share/OVMF. Maybe if this info is relevant, here is the contents of that dir:

$ tree /usr/share/OVMF

/usr/share/OVMF
├── aarch64
│   ├── QEMU_CODE.fd
│   ├── QEMU_EFI.fd
│   └── QEMU_VARS.fd
├── arm
│   ├── QEMU_CODE.fd
│   ├── QEMU_EFI.fd
│   └── QEMU_VARS.fd
├── ia32
│   ├── OVMF.4m.fd
│   ├── OVMF.fd
│   ├── OVMF_CODE.4m.fd
│   ├── OVMF_CODE.csm.4m.fd
│   ├── OVMF_CODE.csm.fd
│   ├── OVMF_CODE.fd
│   ├── OVMF_CODE.secboot.4m.fd
│   ├── OVMF_CODE.secboot.fd
│   ├── OVMF_VARS.4m.fd
│   └── OVMF_VARS.fd
└── x64
    ├── MICROVM.4m.fd
    ├── MICROVM.fd
    ├── OVMF.4m.fd
    ├── OVMF.fd
    ├── OVMF_CODE.4m.fd
    ├── OVMF_CODE.csm.4m.fd
    ├── OVMF_CODE.csm.fd
    ├── OVMF_CODE.fd
    ├── OVMF_CODE.secboot.4m.fd
    ├── OVMF_CODE.secboot.fd
    ├── OVMF_VARS.4m.fd
    └── OVMF_VARS.fd

Linking all of the files from x64 to appear in /usr/share/OVMF via ln ./x64/* ./ didn't help.

I know that AUR is not supported officially, but I really don't want to install and setup snap, so maybe you have any ideas how to debug or fix this error?

[townsend2010]

Hi @nikelborm!

Yeah, this isn't a supported way of running Multipass. That said, if you could provide some logs of how multipassd is launching the instance, I might be able to quickly think of a solution to help you out :wink:

[nikelborm]

Hi, thank very much for your desire to help :)

Unfortunately logs don't seem to be very useful

$ multipass launch -n foo1 -vvvv

[2024-03-19T04:04:40.039] [trace] [url downloader] Found https://codeload.github.com/canonical/multipass-blueprints/zip/refs/heads/main in cache: true
[2024-03-19T04:04:40.042] [debug] [blueprint provider] Loading "anbox-cloud-appliance" v1
[2024-03-19T04:04:40.043] [debug] [blueprint provider] Loading "charm-dev" v1
[2024-03-19T04:04:40.044] [debug] [blueprint provider] Loading "docker" v1
[2024-03-19T04:04:40.045] [debug] [blueprint provider] Loading "jellyfin" v1
[2024-03-19T04:04:40.046] [debug] [blueprint provider] Loading "minikube" v1
[2024-03-19T04:04:40.047] [debug] [blueprint provider] Loading "ros-noetic" v1
[2024-03-19T04:04:40.048] [debug] [blueprint provider] Loading "ros2-humble" v1
[2024-03-19T04:04:40.133] [debug] [qemu-system-x86_64] [208121] started: qemu-system-x86_64 --version
[2024-03-19T04:04:40.210] [debug] [qemu-img] [208125] started: qemu-img info /var/lib/multipassd/.cache/multipassd/vault/images/jammy-20240308/ubuntu-22.04-server-cloudimg-amd64.img
[2024-03-19T04:04:40.241] [debug] [qemu-img] [208130] started: qemu-img resize /var/lib/multipassd/.local/share/multipassd/vault/instances/foo1/ubuntu-22.04-server-cloudimg-amd64.img 5368709120
[2024-03-19T04:04:40.279] [debug] [qemu-img] [208134] started: qemu-img snapshot -l /var/lib/multipassd/.local/share/multipassd/vault/instances/foo1/ubuntu-22.04-server-cloudimg-amd64.img
[2024-03-19T04:04:40.304] [debug] [qemu-img] [208138] started: qemu-img amend -o compat=1.1 /var/lib/multipassd/.local/share/multipassd/vault/instances/foo1/ubuntu-22.04-server-cloudimg-amd64.img
[2024-03-19T04:04:40.346] [debug] [foo1] process working dir ''
[2024-03-19T04:04:40.346] [info] [foo1] process program 'qemu-system-x86_64'
[2024-03-19T04:04:40.346] [info] [foo1] process arguments '-bios, OVMF.fd, --enable-kvm, -cpu, host, -nic, tap,ifname=tap-34ff12516e5,script=no,downscript=no,model=virtio-net-pci,mac=52:54:00:dc:38:3e, -device, virtio-scsi-pci,id=scsi0, -drive, file=/var/lib/multipassd/.local/share/multipassd/vault/instances/foo1/ubuntu-22.04-server-cloudimg-amd64.img,if=none,format=qcow2,discard=unmap,id=hda, -device, scsi-hd,drive=hda,bus=scsi0.0, -smp, 1, -m, 1024M, -qmp, stdio, -chardev, null,id=char0, -serial, chardev:char0, -nographic, -cdrom, /var/lib/multipassd/.local/share/multipassd/vault/instances/foo1/cloud-init-config.iso'
[2024-03-19T04:04:40.364] [debug] [qemu-system-x86_64] [208153] started: qemu-system-x86_64 -nographic -dump-vmstate /tmp/multipassd.mQWPfV
[2024-03-19T04:04:40.483] [info] [foo1] process state changed to Starting
[2024-03-19T04:04:40.498] [info] [foo1] process state changed to Running
[2024-03-19T04:04:40.499] [debug] [qemu-system-x86_64] [208163] started: qemu-system-x86_64 -bios OVMF.fd --enable-kvm -cpu host -nic tap,ifname=tap-34ff12516e5,script=no,downscript=no,model=virtio-net-pci,mac=52:54:00:dc:38:3e -device virtio-scsi-pci,id=scsi0 -drive file=/var/lib/multipassd/.local/share/multipassd/vault/instances/foo1/ubuntu-22.04-server-cloudimg-amd64.img,if=none,format=qcow2,discard=unmap,id=hda -device scsi-hd,drive=hda,bus=scsi0.0 -smp 1 -m 1024M -qmp stdio -chardev null,id=char0 -serial chardev:char0 -nographic -cdrom /var/lib/multipassd/.local/share/multipassd/vault/instances/foo1/cloud-init-config.iso
[2024-03-19T04:04:40.499] [info] [foo1] process started
launch failed: The following errors occurred:
qemu: could not load PC BIOS 'OVMF.fd'
foo1: shutdown called while starting

After seeing this I made a hypothesis that -bios OVMF.fd looks like a file path and since I run multipass launch in my home directory it doesn't see it there. So i cd'ed to /usr/share/OVMF/x64 and run multipass launch and my hypothesis was wrong.

Nothing useful in dmesg -H also.

I also tried to apply hack from here and linked all files

cd /usr/share/OVMF/x64
sudo ln ./* /var/lib/libvirt/qemu/nvram/

But i don't know where to apply second part of the hack

[nikelborm]

By the way I have UEFI system with nvme disk and maybe https://github.com/canonical/multipass/issues/3430#issue-2171583617 is related to my problem. Because when I've researched the issue on the internet I happened to see few mentions of that pflash thing. For example here: https://github.com/vagrant-libvirt/vagrant-libvirt/issues/1725#issuecomment-1453761827

[nikelborm]

I hotfixed it. I checked guess of @olberger from here

my guess, qemu won't want to load a BIOS file that's not under '/usr/share/qemu/' ?

I linked all files from /usr/share/OVMF/x64/ to /usr/share/qemu/ like this:

$ cd /usr/share/OVMF/x64

$ ls -la

Octal Permissions  Size User Group Name
0644  .rw-r--r--  4.0Mi root root   MICROVM.4m.fd
0644  .rw-r--r--  2.0Mi root root   MICROVM.fd
0644  .rw-r--r--  4.0Mi root root   OVMF.4m.fd
0644  .rw-r--r--  2.0Mi root root   OVMF.fd
0644  .rw-r--r--  3.5Mi root root   OVMF_CODE.4m.fd
0644  .rw-r--r--  3.5Mi root root   OVMF_CODE.csm.4m.fd
0644  .rw-r--r--  1.9Mi root root   OVMF_CODE.csm.fd
0644  .rw-r--r--  1.9Mi root root   OVMF_CODE.fd
0644  .rw-r--r--  3.5Mi root root   OVMF_CODE.secboot.4m.fd
0644  .rw-r--r--  1.9Mi root root   OVMF_CODE.secboot.fd
0644  .rw-r--r--  528Ki root root   OVMF_VARS.4m.fd
0644  .rw-r--r--  128Ki root root   OVMF_VARS.fd

$ ln ./* /usr/share/qemu/

'/usr/share/qemu/MICROVM.4m.fd' => './MICROVM.4m.fd'
'/usr/share/qemu/MICROVM.fd' => './MICROVM.fd'
'/usr/share/qemu/OVMF.4m.fd' => './OVMF.4m.fd'
'/usr/share/qemu/OVMF.fd' => './OVMF.fd'
'/usr/share/qemu/OVMF_CODE.4m.fd' => './OVMF_CODE.4m.fd'
'/usr/share/qemu/OVMF_CODE.csm.4m.fd' => './OVMF_CODE.csm.4m.fd'
'/usr/share/qemu/OVMF_CODE.csm.fd' => './OVMF_CODE.csm.fd'
'/usr/share/qemu/OVMF_CODE.fd' => './OVMF_CODE.fd'
'/usr/share/qemu/OVMF_CODE.secboot.4m.fd' => './OVMF_CODE.secboot.4m.fd'
'/usr/share/qemu/OVMF_CODE.secboot.fd' => './OVMF_CODE.secboot.fd'
'/usr/share/qemu/OVMF_VARS.4m.fd' => './OVMF_VARS.4m.fd'
'/usr/share/qemu/OVMF_VARS.fd' => './OVMF_VARS.fd'

And now multipass is able to launch VMs fine

[nikelborm]

By the way /usr/share/OVMF/ is just a link to /usr/share/edk2/. My hypothesis is that multipass looks into /usr/share/qemu/ for that bios file. And instead (or not instead, but also) it should look into /usr/share/edk2/ or /usr/share/edk2/$currentArchitecture or /usr/share/OVMF/$currentArchitecture etc.

Also I found quite interesting .json files in /usr/share/qemu/firmware:

$ grep -rin3 x64

50-edk2-ovmf-x86_64-secure-4m.json-1-{
50-edk2-ovmf-x86_64-secure-4m.json:2:    "description": "x64 UEFI for x86_64, with Secure Boot and SMM, 4MB FD",
50-edk2-ovmf-x86_64-secure-4m.json-3-    "interface-types": [
50-edk2-ovmf-x86_64-secure-4m.json-4-        "uefi"
50-edk2-ovmf-x86_64-secure-4m.json-5-    ],
50-edk2-ovmf-x86_64-secure-4m.json-6-    "mapping": {
50-edk2-ovmf-x86_64-secure-4m.json-7-        "device": "flash",
50-edk2-ovmf-x86_64-secure-4m.json-8-        "executable": {
50-edk2-ovmf-x86_64-secure-4m.json:9:            "filename": "/usr/share/edk2/x64/OVMF_CODE.secboot.4m.fd",
50-edk2-ovmf-x86_64-secure-4m.json-10-            "format": "raw"
50-edk2-ovmf-x86_64-secure-4m.json-11-        },
50-edk2-ovmf-x86_64-secure-4m.json-12-        "nvram-template": {
50-edk2-ovmf-x86_64-secure-4m.json:13:            "filename": "/usr/share/edk2/x64/OVMF_VARS.4m.fd",
50-edk2-ovmf-x86_64-secure-4m.json-14-            "format": "raw"
50-edk2-ovmf-x86_64-secure-4m.json-15-        }
50-edk2-ovmf-x86_64-secure-4m.json-16-    },
--
50-edk2-ovmf-x86_64-secure.json-1-{
50-edk2-ovmf-x86_64-secure.json:2:    "description": "x64 UEFI for x86_64, with Secure Boot and SMM",
50-edk2-ovmf-x86_64-secure.json-3-    "interface-types": [
50-edk2-ovmf-x86_64-secure.json-4-        "uefi"
50-edk2-ovmf-x86_64-secure.json-5-    ],
50-edk2-ovmf-x86_64-secure.json-6-    "mapping": {
50-edk2-ovmf-x86_64-secure.json-7-        "device": "flash",
50-edk2-ovmf-x86_64-secure.json-8-        "executable": {
50-edk2-ovmf-x86_64-secure.json:9:            "filename": "/usr/share/edk2/x64/OVMF_CODE.secboot.fd",
50-edk2-ovmf-x86_64-secure.json-10-            "format": "raw"
50-edk2-ovmf-x86_64-secure.json-11-        },
50-edk2-ovmf-x86_64-secure.json-12-        "nvram-template": {
50-edk2-ovmf-x86_64-secure.json:13:            "filename": "/usr/share/edk2/x64/OVMF_VARS.fd",
50-edk2-ovmf-x86_64-secure.json-14-            "format": "raw"
50-edk2-ovmf-x86_64-secure.json-15-        }
50-edk2-ovmf-x86_64-secure.json-16-    },
--
60-edk2-ovmf-microvm-4m.json-5-    ],
60-edk2-ovmf-microvm-4m.json-6-    "mapping": {
60-edk2-ovmf-microvm-4m.json-7-        "device": "memory",
60-edk2-ovmf-microvm-4m.json:8:        "filename": "/usr/share/edk2/x64/MICROVM.4m.fd"
60-edk2-ovmf-microvm-4m.json-9-    },
60-edk2-ovmf-microvm-4m.json-10-    "targets": [
60-edk2-ovmf-microvm-4m.json-11-        {
--
60-edk2-ovmf-microvm.json-5-    ],
60-edk2-ovmf-microvm.json-6-    "mapping": {
60-edk2-ovmf-microvm.json-7-        "device": "memory",
60-edk2-ovmf-microvm.json:8:        "filename": "/usr/share/edk2/x64/MICROVM.fd"
60-edk2-ovmf-microvm.json-9-    },
60-edk2-ovmf-microvm.json-10-    "targets": [
60-edk2-ovmf-microvm.json-11-        {
--
60-edk2-ovmf-x86_64-4m.json-1-{
60-edk2-ovmf-x86_64-4m.json:2:    "description": "x64 UEFI for x86_64, 4MB FD",
60-edk2-ovmf-x86_64-4m.json-3-    "interface-types": [
60-edk2-ovmf-x86_64-4m.json-4-        "uefi"
60-edk2-ovmf-x86_64-4m.json-5-    ],
60-edk2-ovmf-x86_64-4m.json-6-    "mapping": {
60-edk2-ovmf-x86_64-4m.json-7-        "device": "flash",
60-edk2-ovmf-x86_64-4m.json-8-        "executable": {
60-edk2-ovmf-x86_64-4m.json:9:            "filename": "/usr/share/edk2/x64/OVMF_CODE.4m.fd",
60-edk2-ovmf-x86_64-4m.json-10-            "format": "raw"
60-edk2-ovmf-x86_64-4m.json-11-        },
60-edk2-ovmf-x86_64-4m.json-12-        "nvram-template": {
60-edk2-ovmf-x86_64-4m.json:13:            "filename": "/usr/share/edk2/x64/OVMF_VARS.4m.fd",
60-edk2-ovmf-x86_64-4m.json-14-            "format": "raw"
60-edk2-ovmf-x86_64-4m.json-15-        }
60-edk2-ovmf-x86_64-4m.json-16-    },
--
60-edk2-ovmf-x86_64.json-1-{
60-edk2-ovmf-x86_64.json:2:    "description": "x64 UEFI for x86_64",
60-edk2-ovmf-x86_64.json-3-    "interface-types": [
60-edk2-ovmf-x86_64.json-4-        "uefi"
60-edk2-ovmf-x86_64.json-5-    ],
60-edk2-ovmf-x86_64.json-6-    "mapping": {
60-edk2-ovmf-x86_64.json-7-        "device": "flash",
60-edk2-ovmf-x86_64.json-8-        "executable": {
60-edk2-ovmf-x86_64.json:9:            "filename": "/usr/share/edk2/x64/OVMF_CODE.fd",
60-edk2-ovmf-x86_64.json-10-            "format": "raw"
60-edk2-ovmf-x86_64.json-11-        },
60-edk2-ovmf-x86_64.json-12-        "nvram-template": {
60-edk2-ovmf-x86_64.json:13:            "filename": "/usr/share/edk2/x64/OVMF_VARS.fd",
60-edk2-ovmf-x86_64.json-14-            "format": "raw"
60-edk2-ovmf-x86_64.json-15-        }
60-edk2-ovmf-x86_64.json-16-    },
--
70-edk2-ovmf-x86_64-csm-4m.json-1-{
70-edk2-ovmf-x86_64-csm-4m.json:2:    "description": "x64 UEFI for x86_64, with CSM support, 4MB FD",
70-edk2-ovmf-x86_64-csm-4m.json-3-    "interface-types": [
70-edk2-ovmf-x86_64-csm-4m.json-4-        "uefi"
70-edk2-ovmf-x86_64-csm-4m.json-5-    ],
70-edk2-ovmf-x86_64-csm-4m.json-6-    "mapping": {
70-edk2-ovmf-x86_64-csm-4m.json-7-        "device": "flash",
70-edk2-ovmf-x86_64-csm-4m.json-8-        "executable": {
70-edk2-ovmf-x86_64-csm-4m.json:9:            "filename": "/usr/share/edk2/x64/OVMF_CODE.csm.4m.fd",
70-edk2-ovmf-x86_64-csm-4m.json-10-            "format": "raw"
70-edk2-ovmf-x86_64-csm-4m.json-11-        },
70-edk2-ovmf-x86_64-csm-4m.json-12-        "nvram-template": {
70-edk2-ovmf-x86_64-csm-4m.json:13:            "filename": "/usr/share/edk2/x64/OVMF_VARS.4m.fd",
70-edk2-ovmf-x86_64-csm-4m.json-14-            "format": "raw"
70-edk2-ovmf-x86_64-csm-4m.json-15-        }
70-edk2-ovmf-x86_64-csm-4m.json-16-    },
--
70-edk2-ovmf-x86_64-csm.json-1-{
70-edk2-ovmf-x86_64-csm.json:2:    "description": "x64 UEFI for x86_64, with CSM support",
70-edk2-ovmf-x86_64-csm.json-3-    "interface-types": [
70-edk2-ovmf-x86_64-csm.json-4-        "uefi"
70-edk2-ovmf-x86_64-csm.json-5-    ],
70-edk2-ovmf-x86_64-csm.json-6-    "mapping": {
70-edk2-ovmf-x86_64-csm.json-7-        "device": "flash",
70-edk2-ovmf-x86_64-csm.json-8-        "executable": {
70-edk2-ovmf-x86_64-csm.json:9:            "filename": "/usr/share/edk2/x64/OVMF_CODE.csm.fd",
70-edk2-ovmf-x86_64-csm.json-10-            "format": "raw"
70-edk2-ovmf-x86_64-csm.json-11-        },
70-edk2-ovmf-x86_64-csm.json-12-        "nvram-template": {
70-edk2-ovmf-x86_64-csm.json:13:            "filename": "/usr/share/edk2/x64/OVMF_VARS.fd",
70-edk2-ovmf-x86_64-csm.json-14-            "format": "raw"
70-edk2-ovmf-x86_64-csm.json-15-        }
70-edk2-ovmf-x86_64-csm.json-16-    },

$ ls -la

Octal Permissions Size User Group Name
0644  .rw-r--r--   746 root :      50-edk2-ovmf-i386-secure-4m.json
0644  .rw-r--r--   732 root :      50-edk2-ovmf-i386-secure.json
0644  .rw-r--r--   766 root :      50-edk2-ovmf-x86_64-secure-4m.json
0644  .rw-r--r--   752 root :      50-edk2-ovmf-x86_64-secure.json
0644  .rw-r--r--   622 root :      60-edk2-aarch64.json
0644  .rw-r--r--   607 root :      60-edk2-arm.json
0644  .rw-r--r--   696 root :      60-edk2-ovmf-i386-4m.json
0644  .rw-r--r--   682 root :      60-edk2-ovmf-i386.json
0644  .rw-r--r--   399 root :      60-edk2-ovmf-microvm-4m.json
0644  .rw-r--r--   388 root :      60-edk2-ovmf-microvm.json
0644  .rw-r--r--   716 root :      60-edk2-ovmf-x86_64-4m.json
0644  .rw-r--r--   702 root :      60-edk2-ovmf-x86_64.json
0644  .rw-r--r--   710 root :      70-edk2-ovmf-i386-csm-4m.json
0644  .rw-r--r--   696 root :      70-edk2-ovmf-i386-csm.json
0644  .rw-r--r--   738 root :      70-edk2-ovmf-x86_64-csm-4m.json
0644  .rw-r--r--   724 root :      70-edk2-ovmf-x86_64-csm.json
0644  .rw-r--r--   750 root :      80-edk2-ovmf-ia32-on-x86_64-secure-4m.json
0644  .rw-r--r--   736 root :      80-edk2-ovmf-ia32-on-x86_64-secure.json
0644  .rw-r--r--   700 root :      81-edk2-ovmf-ia32-on-x86_64-4m.json
0644  .rw-r--r--   686 root :      81-edk2-ovmf-ia32-on-x86_64.json
0644  .rw-r--r--   714 root :      82-edk2-ovmf-ia32-on-x86_64-csm-4m.json
0644  .rw-r--r--   700 root :      82-edk2-ovmf-ia32-on-x86_64-csm.json

Maybe if multipass isn't already looking there, it should look into those configs to get proper paths to OVMF.fd files.

I don't know if the issue depends on OVMF.fd exclusively or will it throw different error when I link just OVMF.fd, because i just linked entire batch of them from /usr/share/OVMF/x64 and it just works, which is enough for me. Although if there are better solution, I would love to hear it.

[nikelborm]

Or maybe it is a packaging issue on the side of AUR package. I don't really know

[townsend2010]

Hi @nikelborm!

Glad you got it working! For the record, when building Multipass from source as you did, Multipass only uses the packaged qemu, so I say this is an issue in the qemu AUR package. Since this is working now (and not really a supported configuration), I'm going to close this. Thanks!

[nikelborm]

Yeah, I reported all of this mostly out of the logic that people who will face the same problem in the future will also look for existing issues here and will find this one

[gdamjan]

isn't -bios OVMF.fd try to mutate the OVMF.fd file?