search

lima-vm / lima

Linux virtual machines, with a focus on running containers
https://lima-vm.io/
Apache License 2.0
15.38k stars 603 forks source link

[M1 macOS] Help wanted for testing Lima with QEMU@master #713

Closed AkihiroSuda closed 2 years ago

AkihiroSuda commented 2 years ago

Help wanted for testing https://github.com/lima-vm/lima/pull/703 on M1 macOS

  1. Install Lima v0.9.0-beta.0 or later (brew install --HEAD lima, or git clone https://github.com/lima-vm/lima.git && cd lima && make && make install)
  2. Install QEMU v6.2.0 (brew install qemu)
  3. Create instances with the following memory: 8 GiB, 4 GiB, 2 GiB. 
    
$ limactl start
? Creating an instance "default"  [Use arrows to move, type to filter]
Proceed with the current configuration
> Open an editor to review or modify the current configuration
Choose another example (docker, podman, archlinux, fedora, ...)
Exit

(Change "memory: null" to "memory: 8 GiB" in the editor, and confirm that the instance works) $ limactl delete -f defaut


4. Install QEMU master (`brew install --HEAD qemu`)
5. Repeat the step 3
junnplus commented 2 years ago

I work on M1 macOS, I will take a look.

junnplus commented 2 years ago 
[N] ❯ limactl --version
limactl version HEAD-3a6502a

[N] ❯ qemu-system-aarch64 --version
QEMU emulator version 6.2.0
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers

Work successfully on 8 GiB, 4 GiB, 2 GiB.

[N] ❯ ps aux|grep qemu
jun              90698   1.7  9.6 412361008 3232432 s021  S    10:23PM   1:13.15 /opt/homebrew/bin/qemu-system-aarch64 -cpu host -machine virt,accel=hvf,highmem=off -smp 4,sockets=1,cores=4,threads=1 -m 2048 -drive if=pflash,format=raw,readonly=on,file=/opt/homebrew/share/qemu/edk2-aarch64-code.fd -boot order=c,splash-time=0,menu=on -drive file=/Users/jun/.lima/test2/diffdisk,if=virtio -cdrom /Users/jun/.lima/test2/cidata.iso -netdev user,id=net0,net=192.168.5.0/24,dhcpstart=192.168.5.15,hostfwd=tcp:127.0.0.1:62708-:22 -device virtio-net-pci,netdev=net0,mac=52:55:55:7f:52:9d -device virtio-rng-pci -display none -vga none -device ramfb -device qemu-xhci,id=usb-bus -device usb-kbd,bus=usb-bus.0 -device usb-mouse,bus=usb-bus.0 -parallel none -chardev socket,id=char-serial,path=/Users/jun/.lima/test2/serial.sock,server=on,wait=off,logfile=/Users/jun/.lima/test2/serial.log -serial chardev:char-serial -chardev socket,id=char-qmp,path=/Users/jun/.lima/test2/qmp.sock,server=on,wait=off -qmp chardev:char-qmp -name lima-test2 -pidfile /Users/jun/.lima/test2/qemu.pid
jun              90439   1.4  9.5 414432144 3184464 s020  S    10:22PM   1:13.54 /opt/homebrew/bin/qemu-system-aarch64 -cpu host -machine virt,accel=hvf,highmem=off -smp 4,sockets=1,cores=4,threads=1 -m 4096 -drive if=pflash,format=raw,readonly=on,file=/opt/homebrew/share/qemu/edk2-aarch64-code.fd -boot order=c,splash-time=0,menu=on -drive file=/Users/jun/.lima/test1/diffdisk,if=virtio -cdrom /Users/jun/.lima/test1/cidata.iso -netdev user,id=net0,net=192.168.5.0/24,dhcpstart=192.168.5.15,hostfwd=tcp:127.0.0.1:62662-:22 -device virtio-net-pci,netdev=net0,mac=52:55:55:9a:04:29 -device virtio-rng-pci -display none -vga none -device ramfb -device qemu-xhci,id=usb-bus -device usb-kbd,bus=usb-bus.0 -device usb-mouse,bus=usb-bus.0 -parallel none -chardev socket,id=char-serial,path=/Users/jun/.lima/test1/serial.sock,server=on,wait=off,logfile=/Users/jun/.lima/test1/serial.log -serial chardev:char-serial -chardev socket,id=char-qmp,path=/Users/jun/.lima/test1/qmp.sock,server=on,wait=off -qmp chardev:char-qmp -name lima-test1 -pidfile /Users/jun/.lima/test1/qemu.pid
jun              90047   0.3  9.9 418637776 3320944 s020  S    10:20PM   1:07.94 /opt/homebrew/bin/qemu-system-aarch64 -cpu host -machine virt,accel=hvf,highmem=off -smp 4,sockets=1,cores=4,threads=1 -m 8192 -drive if=pflash,format=raw,readonly=on,file=/opt/homebrew/share/qemu/edk2-aarch64-code.fd -boot order=c,splash-time=0,menu=on -drive file=/Users/jun/.lima/test/diffdisk,if=virtio -cdrom /Users/jun/.lima/test/cidata.iso -netdev user,id=net0,net=192.168.5.0/24,dhcpstart=192.168.5.15,hostfwd=tcp:127.0.0.1:62556-:22 -device virtio-net-pci,netdev=net0,mac=52:55:55:fa:1b:77 -device virtio-rng-pci -display none -vga none -device ramfb -device qemu-xhci,id=usb-bus -device usb-kbd,bus=usb-bus.0 -device usb-mouse,bus=usb-bus.0 -parallel none -chardev socket,id=char-serial,path=/Users/jun/.lima/test/serial.sock,server=on,wait=off,logfile=/Users/jun/.lima/test/serial.log -serial chardev:char-serial -chardev socket,id=char-qmp,path=/Users/jun/.lima/test/qmp.sock,server=on,wait=off -qmp chardev:char-qmp -name lima-test -pidfile /Users/jun/.lima/test/qemu.pid
junnplus commented 2 years ago 
[N] ❯ qemu-system-aarch64 --version
QEMU emulator version 6.2.50 (v6.2.0-2357-g2048c4eba2)
Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers

Unable to start the instance, the system crashes directly. :(

{"level":"debug","msg":"qCmd.Args: [/opt/homebrew/bin/qemu-system-aarch64 -cpu host -machine virt,accel=hvf -smp 4,sockets=1,cores=4,threads=1 -m 4096 -drive if=pflash,format=raw,readonly=on,file=/opt/homebrew/share/qemu/edk2-aarch64-code.fd -boot order=c,splash-time=0,menu=on -drive file=/Users/jun/.lima/vm/diffdisk,if=virtio -cdrom /Users/jun/.lima/vm/cidata.iso -netdev user,id=net0,net=192.168.5.0/24,dhcpstart=192.168.5.15,hostfwd=tcp:127.0.0.1:53432-:22 -device virtio-net-pci,netdev=net0,mac=52:55:55:fe:57:7d -device virtio-rng-pci -display none -vga none -device ramfb -device qemu-xhci,id=usb-bus -device usb-kbd,bus=usb-bus.0 -device usb-mouse,bus=usb-bus.0 -parallel none -chardev socket,id=char-serial,path=/Users/jun/.lima/vm/serial.sock,server=on,wait=off,logfile=/Users/jun/.lima/vm/serial.log -serial chardev:char-serial -chardev socket,id=char-qmp,path=/Users/jun/.lima/vm/qmp.sock,server=on,wait=off -qmp chardev:char-qmp -name lima-vm -pidfile /Users/jun/.lima/vm/qemu.pid]","time":"2022-03-10T22:54:48+08:00"}

masOS error report:

panic(cpu 3 caller 0xfffffe001c0f6a58): vm_fault() KERN_FAILURE from guest fault on state 0xfffffe6019168000 @sleh.c:3091
Debugger message: panic
Memory ID: 0x6
OS release type: User
OS version: 21A559
AkihiroSuda commented 2 years ago

Thank you @Junnplus for analysis.

panic(cpu 3 caller 0xfffffe001c0f6a58): vm_fault() KERN_FAILURE from guest fault on state 0xfffffe6019168000 @sleh.c:3091

Could you report this to the QEMU upstream https://gitlab.com/qemu-project/qemu/-/issues , and also Apple, maybe?

chancez commented 2 years ago

I also got a panic.

(⎈ |rancher-desktop:default) ~ ❯❯❯ limactl --version                                                                                                                                                                                                                                                                                            ✘ 1 
limactl version HEAD-1164273

(⎈ |rancher-desktop:default) ~ ❯❯❯ qemu-system-aarch64 --version
QEMU emulator version 6.2.50 (v6.2.0-2380-g1416688c53)
Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers
panic(cpu 3 caller 0xfffffe001db6ea58): vm_fault() KERN_FAILURE from guest fault on state 0xfffffe6032c98000 @sleh.c:3091
Debugger message: panic
Memory ID: 0x6
OS release type: User
OS version: 21A559
Kernel version: Darwin Kernel Version 21.1.0: Wed Oct 13 17:33:01 PDT 2021; root:xnu-8019.41.5~1/RELEASE_ARM64_T6000

I'll file a bug in qemu upstream and send my problem report to apple.

chancez commented 2 years ago

Qemu issue: https://gitlab.com/qemu-project/qemu/-/issues/903

christhomas commented 2 years ago

I've got an m1 max and I can't even run nerdctl cause it says that it's not installed and I'm unsure what to do next, I've filed a bug ticket about it

junnplus commented 2 years ago

I've got an m1 max and I can't even run nerdctl cause it says that it's not installed and I'm unsure what to do next, I've filed a bug ticket about it

@christhomas It has nothing to do with this issue.

AkihiroSuda commented 2 years ago

panic(cpu 3 caller 0xfffffe001db6ea58): vm_fault() KERN_FAILURE from guest fault on state 0xfffffe6032c98000 @sleh.c:3091

Is this still an issue with macOS 12.3? Has anyone reported to Apple?

jandubois commented 2 years ago

I only have access to an M1 mini and not a pro/max chip, so can't test it myself (the qemu bug mentions that the M1 chip is unaffected)

chancez commented 2 years ago

I'll test with 12.3 and report back. You guys can always ping me on the rancher slack if you need testing on a m1 pro.

chancez commented 2 years ago

No crash on MacOS 12.3 w/ 8GiB memory using latest brewed qemu 6.2 (6.2.0_1) and HEAD Lima. 

(⎈ |rancher-desktop:default) ~ ❯❯❯ sw_vers
ProductName:    macOS
ProductVersion: 12.3
BuildVersion:   21E230
(⎈ |rancher-desktop:default) ~ ❯❯❯ uname -a
Darwin dune.local 21.4.0 Darwin Kernel Version 21.4.0: Mon Feb 21 20:35:58 PST 2022; root:xnu-8020.101.4~2/RELEASE_ARM64_T6000 arm64
(⎈ |rancher-desktop:default) ~ ❯❯❯ limactl --version
limactl version HEAD-250e6b9
(⎈ |rancher-desktop:default) ~ ❯❯❯ qemu-system-aarch64 --version
QEMU emulator version 6.2.0
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers
(⎈ |rancher-desktop:default) ~ ❯❯❯ ls -lah `which qemu-system-aarch64`
lrwxr-xr-x  1 chancezibolski  admin    46B Mar 14 08:17 /opt/homebrew/bin/qemu-system-aarch64 -> ../Cellar/qemu/6.2.0_1/bin/qemu-system-aarch64
(⎈ |rancher-desktop:default) ~ ❯❯❯ limactl delete -f default
WARN[0000] Ignoring non-existent instance "default"     
(⎈ |rancher-desktop:default) ~ ❯❯❯ limactl start
? Creating an instance "default" Open an editor to review or modify the current configuration
INFO[0123] Attempting to download the image from "https://cloud-images.ubuntu.com/releases/21.10/release-20220201/ubuntu-21.10-server-cloudimg-arm64.img"  digest="sha256:1b5b3fe616e1eea4176049d434a360344a7d471f799e151190f21b0a27f0b424"
INFO[0123] Using cache "/Users/chancezibolski/Library/Caches/lima/download/by-url-sha256/7b13036b555773451d0296e74b7f41fd44c96679980ea5769b2439419fc679e3/data" 
INFO[0123] Attempting to download the nerdctl archive from "https://github.com/containerd/nerdctl/releases/download/v0.17.1/nerdctl-full-0.17.1-linux-arm64.tar.gz"  digest="sha256:b773a0db178af9d0963b7c84df88ee434e0c1986fe7491dc1de3231e071e3921"
INFO[0123] Using cache "/Users/chancezibolski/Library/Caches/lima/download/by-url-sha256/ec04f07c11877cda079581d4ac1865088c83d3c1e8be89b1adb61756f716ad77/data" 
INFO[0124] [hostagent] Starting QEMU (hint: to watch the boot progress, see "/Users/chancezibolski/.lima/default/serial.log") 
INFO[0124] SSH Local Port: 60022                        
INFO[0124] [hostagent] Waiting for the essential requirement 1 of 5: "ssh" 
INFO[0179] [hostagent] The essential requirement 1 of 5 is satisfied 
INFO[0179] [hostagent] Waiting for the essential requirement 2 of 5: "user session is ready for ssh" 
INFO[0179] [hostagent] The essential requirement 2 of 5 is satisfied 
INFO[0179] [hostagent] Waiting for the essential requirement 3 of 5: "sshfs binary to be installed" 
INFO[0179] [hostagent] The essential requirement 3 of 5 is satisfied 
INFO[0179] [hostagent] Waiting for the essential requirement 4 of 5: "/etc/fuse.conf to contain \"user_allow_other\"" 
INFO[0194] [hostagent] The essential requirement 4 of 5 is satisfied 
INFO[0194] [hostagent] Waiting for the essential requirement 5 of 5: "the guest agent to be running" 
INFO[0194] [hostagent] The essential requirement 5 of 5 is satisfied 
INFO[0194] [hostagent] Mounting "/Users/chancezibolski" 
INFO[0195] [hostagent] Mounting "/tmp/lima"             
INFO[0195] [hostagent] Waiting for the optional requirement 1 of 2: "systemd must be available" 
INFO[0195] [hostagent] Forwarding "/run/lima-guestagent.sock" (guest) to "/Users/chancezibolski/.lima/default/ga.sock" (host) 
INFO[0195] [hostagent] The optional requirement 1 of 2 is satisfied 
INFO[0195] [hostagent] Waiting for the optional requirement 2 of 2: "containerd binaries to be installed" 
INFO[0195] [hostagent] Not forwarding TCP 127.0.0.53:53 
INFO[0195] [hostagent] Not forwarding TCP 0.0.0.0:22    
INFO[0195] [hostagent] Not forwarding TCP [::]:22       
INFO[0198] [hostagent] The optional requirement 2 of 2 is satisfied 
INFO[0198] [hostagent] Waiting for the final requirement 1 of 1: "boot scripts must have finished" 
INFO[0210] [hostagent] The final requirement 1 of 1 is satisfied 
INFO[0210] READY. Run `lima` to open the shell. 
(⎈ |rancher-desktop:default) ~ ❯❯❯ lima
chancezibolski@lima-default:/Users/chancezibolski$ free -h
               total        used        free      shared  buff/cache   available
Mem:           7.7Gi       209Mi       6.3Gi       1.0Mi       1.2Gi       7.4Gi
Swap:             0B          0B          0B

Qemu cmd:

{"level":"debug","msg":"qCmd.Args: [/opt/homebrew/bin/qemu-system-aarch64 -cpu host -machine virt,accel=hvf,highmem=off -smp 4,sockets=1,cores=4,threads=1 -m 8192 -drive if=pflash,format=raw,readonly=on,file=/opt/homebrew/share/qemu/edk2-aarch64-code.fd -boot order=c,splash-time=0,menu=on -drive file=/Users/chancezibolski/.lima/default/diffdisk,if=virtio -cdrom /Users/chancezibolski/.lima/default/cidata.iso -netdev user,id=net0,net=192.168.5.0/24,dhcpstart=192.168.5.15,hostfwd=tcp:127.0.0.1:60022-:22 -device virtio-net-pci,netdev=net0,mac=52:55:55:35:a3:d1 -device virtio-rng-pci -display none -vga none -device ramfb -device qemu-xhci,id=usb-bus -device usb-kbd,bus=usb-bus.0 -device usb-mouse,bus=usb-bus.0 -parallel none -chardev socket,id=char-serial,path=/Users/chancezibolski/.lima/default/serial.sock,server=on,wait=off,logfile=/Users/chancezibolski/.lima/default/serial.log -serial chardev:char-serial -chardev socket,id=char-qmp,path=/Users/chancezibolski/.lima/default/qmp.sock,server=on,wait=off -qmp chardev:char-qmp -name lima-default -pidfile /Users/chancezibolski/.lima/default/qemu.pid]","time":"2022-03-17T10:58:59-07:00"}
chancez commented 2 years ago

Panic on 12.3 w/ qemu HEAD:

panic(cpu 2 caller 0xfffffe001546d814): vm_fault() KERN_FAILURE from guest fault on state 0xfffffe6032ae4000 @sleh.c:3117
Debugger message: panic
Memory ID: 0x6
OS release type: User
OS version: 21E230
Kernel version: Darwin Kernel Version 21.4.0: Mon Feb 21 20:35:58 PST 2022; root:xnu-8020.101.4~2/RELEASE_ARM64_T6000
Fileset Kernelcache UUID: 65F083231F3684E89C62E16ACD8BF64E
Kernel UUID: 178AA913-FCA9-33FD-A81C-DF08315F458D
network-shark commented 2 years ago

For me it works 👍 , but I did not update qemu6.2.0 to 6.2.0_1 .

╰─ limactl list
NAME       STATUS     SSH                ARCH       CPUS    MEMORY    DISK      DIR
default    Running    127.0.0.1:60022    aarch64    4       4GiB      100GiB    /Users/sharky/.lima/default

╰─ limactl -v
limactl version HEAD-50beb56
AkihiroSuda commented 2 years ago

Thanks all for testing. Can anybody check whether https://lists.gnu.org/archive/html/qemu-devel/2022-03/msg00494.html fixes the host panic on M1 (Pro?)?

network-shark commented 2 years ago

How can I accomplish it ?

Thanks all for testing. Can anybody check whether https://lists.gnu.org/archive/html/qemu-devel/2022-03/msg00494.html fixes the host panic on M1 (Pro?)?

How can I use the patches ?

chancez commented 2 years ago

I'll test the patches. I checked the qemu docs and they have this project called patchew which makes patches available on GitHub:

Here's how I'm testing:

brew unlink qemu
brew edit qemu

then remove all the patches , and modify the following line:

  head "https://git.qemu.org/git/qemu.git", branch: "master"

And update it to be:

  head "https://github.com/patchew-project/qemu.git", tag: "patchew/20220302130417.18551-1-ubzeme@gmail.com"

Then install the qemu like this:

export HOMEBREW_NO_AUTO_UPDATE=1
brew install --build-from-source --HEAD qemu

You can also just checkout the homebrew branch I created with these changes if you like: https://github.com/chancez/homebrew-core/tree/qemu_hvf_patched

cd "$(brew --repository homebrew/core)"
git remote add chancez git@github.com:chancez/homebrew-core.git
git fetch chancez
git checkout chancez/qemu_hvf_patched
export HOMEBREW_NO_AUTO_UPDATE=1
brew unlink qemu
brew install --build-from-source --HEAD qemu
chancez commented 2 years ago

Got another panic with the above patches

(⎈ |rancher-desktop:default) ~ ❯❯❯ qemu-system-aarch64 --version
QEMU emulator version 6.2.90 (v6.2.0-2525-g1535ec6d77)
Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers

v6.2.0-2525-g1535ec6d77 the last bit g1535ec6d77, without the leading g, matches the SHA here: https://github.com/patchew-project/qemu/commit/1535ec6d771df68200a85ce388e05ff0a34ffaa3

Here's the panic:

panic(cpu 3 caller 0xfffffe00146f5814): vm_fault() KERN_FAILURE from guest fault on state 0xfffffe61154e4000 @sleh.c:3117
Debugger message: panic
Memory ID: 0x6
OS release type: User
OS version: 21E230
Kernel version: Darwin Kernel Version 21.4.0: Mon Feb 21 20:35:58 PST 2022; root:xnu-8020.101.4~2/RELEASE_ARM64_T6000
Fileset Kernelcache UUID: 65F083231F3684E89C62E16ACD8BF64E
Kernel UUID: 178AA913-FCA9-33FD-A81C-DF08315F458D
iBoot version: iBoot-7459.101.2
chancez commented 2 years ago

According to another user, it sounds like this may be fixed in macOS 12.4 Beta1

https://gitlab.com/qemu-project/qemu/-/issues/903#note_911000975