search

lima-vm / lima

Linux virtual machines, with a focus on running containers
https://lima-vm.io/
Apache License 2.0
15.12k stars 591 forks source link

Some UEFI boots extremely slow (ArchLinux) #1337

Closed merlinscholz closed 1 year ago

merlinscholz commented 1 year ago

Description

Hello, when creating a Lima machine via...

limactl start --name=default template://archlinux

...on an aarch64 host, the boot process is extremely slow. Inspecting the boot log reveals that QEMU is trying to boot from PXE (both IPv4 and IPv6), and HTTP (again IPv4 and IPv6) after that. Since none of those options will work, there are 4 additional timeouts to wait for before booting the actual VM.

Furthermore, a 5 second delay from Press ESC in 1 seconds to skip startup.nsh or any other key to continue. during the UEFI boot is being added.

Are those intended as some kind of fallback options or is this due to a missing QEMU boot configuration? In the boot logs there's also some info on a missing "UEFI Misc device" with a seemingly unique ID, maybe the whole PXE/HTTP stuff is being tried because that device cannot be found properly?

serial.log:

BdsDxe: failed to load Boot0001 "UEFI Misc Device" from VenHw(93E34C7E-B50E-11DF-9223-2443DFD72085,00): Not Found
BdsDxe: failed to load Boot0002 "UEFI Misc Device 2" from PciRoot(0x0)/Pci(0x4,0x0): Not Found
BdsDxe: failed to load Boot0003 "UEFI Misc Device 3" from PciRoot(0x0)/Pci(0x5,0x0): Not Found

>>Start PXE over IPv4.
  PXE-E16: No valid offer received.
BdsDxe: failed to load Boot0004 "UEFI PXEv4 (MAC:5255559F3B22)" from PciRoot(0x0)/Pci(0x1,0x0)/MAC(5255559F3B22,0x1)/IPv4(0.0.0.0,0x0,DHCP,0.0.0.0,0.0.0.0,0.0.0.0): Not Found

>>Start PXE over IPv6.
  PXE-E16: No valid offer received.
BdsDxe: failed to load Boot0005 "UEFI PXEv6 (MAC:5255559F3B22)" from PciRoot(0x0)/Pci(0x1,0x0)/MAC(5255559F3B22,0x1)/IPv6(0000:0000:0000:0000:0000:0000:0000:0000,0x0,Static,0000:0000:0000:0000:0000:0000:0000:0000,0x40,0000:0000:0000:0000:0000:0000:0000:0000): Not Found

>>Start HTTP Boot over IPv4.....
  Error: Could not retrieve NBP file size from HTTP server.

  Error: Server response timeout.
BdsDxe: failed to load Boot0006 "UEFI HTTPv4 (MAC:5255559F3B22)" from PciRoot(0x0)/Pci(0x1,0x0)/MAC(5255559F3B22,0x1)/IPv4(0.0.0.0,0x0,DHCP,0.0.0.0,0.0.0.0,0.0.0.0)/Uri(): Not Found

>>Start HTTP Boot over IPv6.
  Error: Could not retrieve NBP file size from HTTP server.

  Error: Unexpected network error.
BdsDxe: failed to load Boot0007 "UEFI HTTPv6 (MAC:5255559F3B22)" from PciRoot(0x0)/Pci(0x1,0x0)/MAC(5255559F3B22,0x1)/IPv6(0000:0000:0000:0000:0000:0000:0000:0000,0x0,Static,0000:0000:0000:0000:0000:0000:0000:0000,0x40,0000:0000:0000:0000:0000:0000:0000:0000)/Uri(): Not Found
BdsDxe: loading Boot0008 "EFI Internal Shell" from Fv(64074AFE-340A-4BE6-94BA-91B5B4D0F71E)/FvFile(7C04A583-9E3E-4F1C-AD65-E05268D0B4D1)
BdsDxe: starting Boot0008 "EFI Internal Shell" from Fv(64074AFE-340A-4BE6-94BA-91B5B4D0F71E)/FvFile(7C04A583-9E3E-4F1C-AD65-E05268D0B4D1)
UEFI Interactive Shell v2.2
EDK II
UEFI v2.70 (EDK II, 0x00010000)
Mapping table
      FS0: Alias(s):HD0b:;BLK1:
          PciRoot(0x0)/Pci(0x4,0x0)/HD(1,GPT,C73D2392-1833-4FCE-ACF0-1E4A8FF16019,0x800,0x64000)
     BLK4: Alias(s):
          VenHw(93E34C7E-B50E-11DF-9223-2443DFD72085,00)
     BLK0: Alias(s):
          PciRoot(0x0)/Pci(0x4,0x0)
     BLK2: Alias(s):
          PciRoot(0x0)/Pci(0x4,0x0)/HD(2,GPT,C54157EA-CDAA-4834-9C31-B05B50EC04AF,0x64800,0xC79B7DF)
     BLK3: Alias(s):
          PciRoot(0x0)/Pci(0x5,0x0)

Press ESC in 1 seconds to skip startup.nsh or any other key to continue.
Shell> Image root=UUID=7a2dc097-ea12-4671-8951-7ec022c34d57 rw initrd=\initramfs-linux.img
EFI stub: Booting Linux Kernel...
EFI stub: Generating empty DTB
EFI stub: Loaded initrd from command line option
EFI stub: Exiting boot services...

After that, the expected Linux boot begins.

limactl info:

{
    "version": "0.14.2",
    "templates": [
        {
            "name": "almalinux-8",
            "location": "/opt/homebrew/share/lima/examples/almalinux-8.yaml"
        },
        {
            "name": "almalinux-9",
            "location": "/opt/homebrew/share/lima/examples/almalinux-9.yaml"
        },
        {
            "name": "almalinux",
            "location": "/opt/homebrew/share/lima/examples/almalinux.yaml"
        },
        {
            "name": "alpine",
            "location": "/opt/homebrew/share/lima/examples/alpine.yaml"
        },
        {
            "name": "apptainer-rootful",
            "location": "/opt/homebrew/share/lima/examples/apptainer-rootful.yaml"
        },
        {
            "name": "apptainer",
            "location": "/opt/homebrew/share/lima/examples/apptainer.yaml"
        },
        {
            "name": "archlinux",
            "location": "/opt/homebrew/share/lima/examples/archlinux.yaml"
        },
        {
            "name": "buildkit",
            "location": "/opt/homebrew/share/lima/examples/buildkit.yaml"
        },
        {
            "name": "centos-stream-8",
            "location": "/opt/homebrew/share/lima/examples/centos-stream-8.yaml"
        },
        {
            "name": "centos-stream-9",
            "location": "/opt/homebrew/share/lima/examples/centos-stream-9.yaml"
        },
        {
            "name": "centos-stream",
            "location": "/opt/homebrew/share/lima/examples/centos-stream.yaml"
        },
        {
            "name": "debian",
            "location": "/opt/homebrew/share/lima/examples/debian.yaml"
        },
        {
            "name": "default",
            "location": "/opt/homebrew/share/lima/examples/default.yaml"
        },
        {
            "name": "deprecated/centos-7",
            "location": "/opt/homebrew/share/lima/examples/deprecated/centos-7.yaml"
        },
        {
            "name": "docker-rootful",
            "location": "/opt/homebrew/share/lima/examples/docker-rootful.yaml"
        },
        {
            "name": "docker",
            "location": "/opt/homebrew/share/lima/examples/docker.yaml"
        },
        {
            "name": "experimental/9p",
            "location": "/opt/homebrew/share/lima/examples/experimental/9p.yaml"
        },
        {
            "name": "experimental/opensuse-tumbleweed",
            "location": "/opt/homebrew/share/lima/examples/experimental/opensuse-tumbleweed.yaml"
        },
        {
            "name": "experimental/riscv64",
            "location": "/opt/homebrew/share/lima/examples/experimental/riscv64.yaml"
        },
        {
            "name": "experimental/vz",
            "location": "/opt/homebrew/share/lima/examples/experimental/vz.yaml"
        },
        {
            "name": "faasd",
            "location": "/opt/homebrew/share/lima/examples/faasd.yaml"
        },
        {
            "name": "fedora",
            "location": "/opt/homebrew/share/lima/examples/fedora.yaml"
        },
        {
            "name": "k3s",
            "location": "/opt/homebrew/share/lima/examples/k3s.yaml"
        },
        {
            "name": "k8s",
            "location": "/opt/homebrew/share/lima/examples/k8s.yaml"
        },
        {
            "name": "nomad",
            "location": "/opt/homebrew/share/lima/examples/nomad.yaml"
        },
        {
            "name": "opensuse",
            "location": "/opt/homebrew/share/lima/examples/opensuse.yaml"
        },
        {
            "name": "oraclelinux-8",
            "location": "/opt/homebrew/share/lima/examples/oraclelinux-8.yaml"
        },
        {
            "name": "oraclelinux-9",
            "location": "/opt/homebrew/share/lima/examples/oraclelinux-9.yaml"
        },
        {
            "name": "oraclelinux",
            "location": "/opt/homebrew/share/lima/examples/oraclelinux.yaml"
        },
        {
            "name": "podman-rootful",
            "location": "/opt/homebrew/share/lima/examples/podman-rootful.yaml"
        },
        {
            "name": "podman",
            "location": "/opt/homebrew/share/lima/examples/podman.yaml"
        },
        {
            "name": "rocky-8",
            "location": "/opt/homebrew/share/lima/examples/rocky-8.yaml"
        },
        {
            "name": "rocky-9",
            "location": "/opt/homebrew/share/lima/examples/rocky-9.yaml"
        },
        {
            "name": "rocky",
            "location": "/opt/homebrew/share/lima/examples/rocky.yaml"
        },
        {
            "name": "ubuntu-lts",
            "location": "/opt/homebrew/share/lima/examples/ubuntu-lts.yaml"
        },
        {
            "name": "ubuntu",
            "location": "/opt/homebrew/share/lima/examples/ubuntu.yaml"
        },
        {
            "name": "vmnet",
            "location": "/opt/homebrew/share/lima/examples/vmnet.yaml"
        }
    ],
    "defaultTemplate": {
        "vmType": "qemu",
        "arch": "aarch64",
        "images": [
            {
                "location": "https://cloud-images.ubuntu.com/releases/22.10/release-20221201/ubuntu-22.10-server-cloudimg-amd64.img",
                "arch": "x86_64",
                "digest": "sha256:4228fae635160ee2eeebda7b3f466e99729121958c125c6fbefe79178355d09b"
            },
            {
                "location": "https://cloud-images.ubuntu.com/releases/22.10/release-20221201/ubuntu-22.10-server-cloudimg-arm64.img",
                "arch": "aarch64",
                "digest": "sha256:9575dfe9f925ec251a933b88a38c5582a18e9d19495025ac01cb2e217e5f14ca"
            },
            {
                "location": "https://cloud-images.ubuntu.com/releases/22.10/release/ubuntu-22.10-server-cloudimg-amd64.img",
                "arch": "x86_64"
            },
            {
                "location": "https://cloud-images.ubuntu.com/releases/22.10/release/ubuntu-22.10-server-cloudimg-arm64.img",
                "arch": "aarch64"
            }
        ],
        "cpuType": {
            "aarch64": "host",
            "riscv64": "rv64",
            "x86_64": "qemu64"
        },
        "cpus": 4,
        "memory": "4GiB",
        "disk": "100GiB",
        "mounts": [
            {
                "location": "~",
                "mountPoint": "~",
                "writable": false,
                "sshfs": {
                    "cache": true,
                    "followSymlinks": false,
                    "sftpDriver": ""
                },
                "9p": {
                    "securityModel": "none",
                    "protocolVersion": "9p2000.L",
                    "msize": "128KiB",
                    "cache": "fscache"
                }
            },
            {
                "location": "/tmp/lima",
                "mountPoint": "/tmp/lima",
                "writable": true,
                "sshfs": {
                    "cache": true,
                    "followSymlinks": false,
                    "sftpDriver": ""
                },
                "9p": {
                    "securityModel": "none",
                    "protocolVersion": "9p2000.L",
                    "msize": "128KiB",
                    "cache": "mmap"
                }
            }
        ],
        "mountType": "reverse-sshfs",
        "ssh": {
            "localPort": 0,
            "loadDotSSHPubKeys": true,
            "forwardAgent": false,
            "forwardX11": false,
            "forwardX11Trusted": false
        },
        "firmware": {
            "legacyBIOS": false
        },
        "video": {
            "display": "none"
        },
        "containerd": {
            "system": false,
            "user": true,
            "archives": [
                {
                    "location": "https://github.com/containerd/nerdctl/releases/download/v1.1.0/nerdctl-full-1.1.0-linux-amd64.tar.gz",
                    "arch": "x86_64",
                    "digest": "sha256:5440c7b3af63df2ad2c98e185e06a27b4a21eea334b05408e84f8502251d9459"
                },
                {
                    "location": "https://github.com/containerd/nerdctl/releases/download/v1.1.0/nerdctl-full-1.1.0-linux-arm64.tar.gz",
                    "arch": "aarch64",
                    "digest": "sha256:3b613a1be5a24460c44bb93a3609b790ada94e06efd1a86467d45bec7da8b449"
                }
            ]
        },
        "hostResolver": {
            "enabled": true,
            "ipv6": false
        },
        "propagateProxyEnv": true,
        "caCerts": {
            "removeDefaults": false
        },
        "rosetta": {
            "enabled": false,
            "binfmt": false
        }
    },
    "limaHome": "/Users/merlin/.lima",
    "vmTypes": [
        "qemu",
        "vz"
    ]
}
AkihiroSuda commented 1 year ago

Is this specific to archlinux?

merlinscholz commented 1 year ago

So far I've tried Fedora, Alma, Ubuntu, Debian and Rocky, the issue only appeared in Arch. I'll update when I find another problematic OS.

merlinscholz commented 1 year ago

The only relevant detail that makes Arch stand out is that they - as only OS - do not use Grub by default in their cloud images, but rather boot the EFI-compatible kernel directly via EFISTUB

lobshunter commented 1 year ago

I suppose this issue is dup of #858. Author of arch-aarch64 image has described the root cause.

IMHO the reason of such issue is archlinux and archlinux-arm are two separated project. Currently archlinux-arm doesn't provide OS image, so some contributors stand out and build one(work appreciated).

It's not easy for an individual contributor to actively maintain a project. I personally switched to ubuntu instead.