search

astro / microvm.nix

NixOS MicroVMs
https://astro.github.io/microvm.nix/
MIT License
1.24k stars 92 forks source link

cloud-hypervisor broken on aarch64-linux #105

Open mikatammi opened 1 year ago

mikatammi commented 1 year ago

I have a hydra setup which is building microvms. What I see is in log:

Copying a /nix/store
Creating a erofs
mkfs.erofs 1.6
Build completed.mygr3wkmv3d0-gpm-unstable-2020-06-17/share/man/man8/gpm.8.gz ...
Cleaning up store
Copying store erofs into disk
953712+0 records in
953712+0 records out
488300544 bytes (488 MB, 466 MiB) copied, 1.45549 s, 335 MB/s
1+0 records in
1+0 records out
512 bytes copied, 6.8916e-05 s, 7.4 MB/s
Creating partition table
Creating EFI partition

I have waited for days and it never gets past the "Creating EFI partition" step.

With ps ax on the builder machine I can see qemu-kvm that has been running for a long time.

 958966 ?        Ss     0:00 bash -e /nix/store/6xg259477c90a229xwmb53pdfkn6ig3g-default-builder.sh
 959530 ?        S      0:00 /nix/store/vgzggbs5dv4ay3wv0dxcvc9k59zkbd8l-libguestfs-1.48.4/bin/guestfish --add /nix/store/94xchp0c0h8533mnqkdy2i64w9llzrjx-microvm-bootdisk.img run : mkfs fat /dev/sda1
 959553 ?        Sl   1086:46 /nix/store/5isknrpw41bpv4c6qcq230xa6k6kx6kd-qemu-8.0.0/bin/qemu-kvm -global virtio-blk-pci.scsi=off -no-user-config -nodefaults -display none -machine virt,accel=kvm:tcg -cpu cortex-a57 -m 1280 -no-reboot -rtc driftfix=slew -kernel /nix/store/wsb0ls0rr4v1kl5rylx
 959554 ?        S      0:00 /nix/store/vgzggbs5dv4ay3wv0dxcvc9k59zkbd8l-libguestfs-1.48.4/bin/guestfish --add /nix/store/94xchp0c0h8533mnqkdy2i64w9llzrjx-microvm-bootdisk.img run : mkfs fat /dev/sda1

I'd need to add some debug prints or something to see what is the actual command after "Creating EFI partition" that is taking so long.

This only seems to appear on aarch64-linux targets and not x86_64-linux

astro commented 1 year ago

I fear I broke your great aarch64-linux work. If you require something useable right now then use tag v0.3.3.

By now, the bootDisk is only required for the hypervisors that cannot load initrds by themselves. Namely: cloud-hypervisor. Their solution is https://github.com/cloud-hypervisor/rust-hypervisor-firmware which I packaged in time for nixos-23.05.

I used the last stable release 0.4.2 which supports only x86_64-linux. I had trouble packaging an unstable version but we shall try again because it brings support for aarch64 and riscv64.

Do the other hypervisors work for you? They shouldn't require the bootDisk to be built.

mikatammi commented 1 year ago

I have to double check this because looks like with latest versions the builds work again, but I haven't tested the functionality

mikatammi commented 1 year ago

My build is only using QEMU at the moment, didn't try other hypervisors yet

mikatammi commented 1 year ago

https://github.com/astro/microvm.nix/commit/708f09d5a093af670aa4a7ae4d9d0b0a10f31276 I guess this commit fixes this issue for other hypervisors than cloud-hypervisor

astro commented 11 months ago

Since 4f752aec0f666f86fa2b82ba7e5540a687ceeabe we no longer require x86_64-only rust-hypervisor-firmware again.