search

abiosoft / colima

Container runtimes on macOS (and Linux) with minimal setup
MIT License
19.52k stars 395 forks source link

Colima not starting "FATA[0002] error starting vm: error at 'starting': exit status 1" #786

Closed mikaalanwar closed 1 year ago

mikaalanwar commented 1 year ago

Description

Colima does not start on my local workstation. Here are the logs when I try to start colima:

❯ colima start
INFO[0000] starting colima
INFO[0000] runtime: docker
INFO[0000] preparing network ...                         context=vm
WARN[0000] disk size cannot be reduced, ignoring...      context=vm
INFO[0000] starting ...                                  context=vm
> [hostagent] Shutting down the host agent
> [hostagent] failed to exit SSH master
> [hostagent] Shutting down QEMU with ACPI
> [hostagent] failed to open the QMP socket "/Users/mikaalanwar/.lima/colima/qmp.sock", forcibly killing QEMU
> [hostagent] QEMU has already exited
> exiting, status={Running:false Degraded:false Exiting:true Errors:[] SSHLocalPort:0} (hint: see "/Users/mikaalanwar/.lima/colima/ha.stderr.log")
FATA[0002] error starting vm: error at 'starting': exit status 1

Here's my log file:

{"level":"debug","msg":"Creating iso file /Users/mikaalanwar/.lima/colima/cidata.iso","time":"2023-08-25T12:47:46+02:00"}
{"level":"debug","msg":"Using /var/folders/my/cr47js4n3y122yf6bf31vq45dh2ds8/T/diskfs_iso936620864 as workspace","time":"2023-08-25T12:47:46+02:00"}
{"level":"debug","msg":"OpenSSH version 8.6.1 detected","time":"2023-08-25T12:47:47+02:00"}
{"level":"debug","msg":"AES accelerator seems available, prioritizing aes128-gcm@openssh.com and aes256-gcm@openssh.com","time":"2023-08-25T12:47:47+02:00"}
{"level":"debug","msg":"QEMU version 8.1.0 detected","time":"2023-08-25T12:47:47+02:00"}
{"level":"debug","msg":"firmware candidates = [/Users/mikaalanwar/.local/share/qemu/edk2-x86_64-code.fd /Users/mikaalanwar/.colima/_wrapper/4e1b408f843d1c63afbbdcf80c40e4c88d33509f/share/qemu/edk2-x86_64-code.fd /usr/share/OVMF/OVMF_CODE.fd /usr/share/qemu/ovmf-x86_64-code.bin /usr/share/edk2-ovmf/x64/OVMF_CODE.fd]","time":"2023-08-25T12:47:47+02:00"}
{"level":"info","msg":"Starting QEMU (hint: to watch the boot progress, see \"/Users/mikaalanwar/.lima/colima/serial*.log\")","time":"2023-08-25T12:47:47+02:00"}
{"level":"debug","msg":"qCmd.Args: [/Users/mikaalanwar/.colima/_wrapper/4e1b408f843d1c63afbbdcf80c40e4c88d33509f/bin/qemu-system-x86_64 -m 12288 -cpu host,-pdpe1gb -machine q35,accel=hvf -smp 4,sockets=1,cores=4,threads=1 -drive if=pflash,format=raw,readonly=on,file=/Users/mikaalanwar/.colima/_wrapper/4e1b408f843d1c63afbbdcf80c40e4c88d33509f/share/qemu/edk2-x86_64-code.fd -boot order=d,splash-time=0,menu=on -drive file=/Users/mikaalanwar/.lima/colima/basedisk,format=raw,media=cdrom,readonly=on -drive file=/Users/mikaalanwar/.lima/colima/diffdisk,if=virtio,discard=on -drive id=cdrom0,if=none,format=raw,readonly=on,file=/Users/mikaalanwar/.lima/colima/cidata.iso -device virtio-scsi-pci,id=scsi0 -device scsi-cd,bus=scsi0.0,drive=cdrom0 -netdev user,id=net0,net=192.168.5.0/24,dhcpstart=192.168.5.15,hostfwd=tcp:127.0.0.1:51472-:22 -device virtio-net-pci,netdev=net0,mac=52:55:55:20:2f:11 -device virtio-rng-pci -display none -device virtio-vga -device virtio-keyboard-pci -device virtio-mouse-pci -device qemu-xhci,id=usb-bus -parallel none -chardev socket,id=char-serial,path=/Users/mikaalanwar/.lima/colima/serial.sock,server=on,wait=off,logfile=/Users/mikaalanwar/.lima/colima/serial.log -serial chardev:char-serial -chardev socket,id=char-serial-virtio,path=/Users/mikaalanwar/.lima/colima/serialv.sock,server=on,wait=off,logfile=/Users/mikaalanwar/.lima/colima/serialv.log -device virtio-serial-pci,id=virtio-serial0,max_ports=1 -device virtconsole,chardev=char-serial-virtio,id=console0 -chardev socket,id=char-qmp,path=/Users/mikaalanwar/.lima/colima/qmp.sock,server=on,wait=off -qmp chardev:char-qmp -name lima-colima -pidfile /Users/mikaalanwar/.lima/colima/qemu.pid]","time":"2023-08-25T12:47:48+02:00"}
{"level":"info","msg":"Waiting for the essential requirement 1 of 5: \"ssh\"","time":"2023-08-25T12:47:48+02:00"}
{"level":"debug","msg":"executing script \"ssh\"","time":"2023-08-25T12:47:48+02:00"}
{"level":"debug","msg":"executing ssh for script \"ssh\": /usr/bin/ssh [ssh -F /dev/null -o IdentityFile=\"/Users/mikaalanwar/.lima/_config/user\" -o IdentityFile=\"/Users/mikaalanwar/.ssh/id_ed25519\" -o IdentityFile=\"/Users/mikaalanwar/.ssh/mikaalanwar-GitHub\" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NoHostAuthenticationForLocalhost=yes -o GSSAPIAuthentication=no -o PreferredAuthentications=publickey -o Compression=no -o BatchMode=yes -o IdentitiesOnly=yes -o Ciphers=\"^aes128-gcm@openssh.com,aes256-gcm@openssh.com\" -o User=lima -o ControlMaster=auto -o ControlPath=\"/Users/mikaalanwar/.lima/colima/ssh.sock\" -o ControlPersist=5m -p 51472 127.0.0.1 -- /bin/bash]","time":"2023-08-25T12:47:48+02:00"}
{"level":"debug","msg":"stdout=\"\", stderr=\"ssh: connect to host 127.0.0.1 port 51472: Connection refused\\r\\n\", err=failed to execute script \"ssh\": stdout=\"\", stderr=\"ssh: connect to host 127.0.0.1 port 51472: Connection refused\\r\\n\": exit status 255","time":"2023-08-25T12:47:48+02:00"}
{"level":"debug","msg":"qemu[stderr]: qemu-system-x86_64: Error: HV_DENIED","time":"2023-08-25T12:47:48+02:00"}
{"level":"info","msg":"Driver stopped due to error: \"exit status 255\"","time":"2023-08-25T12:47:48+02:00"}
{"level":"info","msg":"Shutting down the host agent","time":"2023-08-25T12:47:48+02:00"}
{"level":"debug","msg":"shutting down the SSH master","time":"2023-08-25T12:47:48+02:00"}
{"level":"debug","msg":"executing ssh for exiting the master: /usr/bin/ssh [ssh -F /dev/null -o IdentityFile=\"/Users/mikaalanwar/.lima/_config/user\" -o IdentityFile=\"/Users/mikaalanwar/.ssh/id_ed25519\" -o IdentityFile=\"/Users/mikaalanwar/.ssh/mikaalanwar-GitHub\" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NoHostAuthenticationForLocalhost=yes -o GSSAPIAuthentication=no -o PreferredAuthentications=publickey -o Compression=no -o BatchMode=yes -o IdentitiesOnly=yes -o Ciphers=\"^aes128-gcm@openssh.com,aes256-gcm@openssh.com\" -o User=lima -o ControlMaster=auto -o ControlPath=\"/Users/mikaalanwar/.lima/colima/ssh.sock\" -o ControlPersist=5m -O exit -p 51472 127.0.0.1]","time":"2023-08-25T12:47:48+02:00"}
{"error":"failed to execute `ssh -O exit -p 51472 127.0.0.1`, out=\"Control socket connect(/Users/mikaalanwar/.lima/colima/ssh.sock): No such file or directory\\r\\n\": exit status 255","level":"warning","msg":"failed to exit SSH master","time":"2023-08-25T12:47:48+02:00"}
{"level":"info","msg":"Shutting down QEMU with ACPI","time":"2023-08-25T12:47:48+02:00"}
{"error":"dial unix /Users/mikaalanwar/.lima/colima/qmp.sock: connect: connection refused","level":"warning","msg":"failed to open the QMP socket \"/Users/mikaalanwar/.lima/colima/qmp.sock\", forcibly killing QEMU","time":"2023-08-25T12:47:48+02:00"}
{"level":"info","msg":"QEMU has already exited","time":"2023-08-25T12:47:48+02:00"}

I have tried the following:

colima stop
colima delete -f
brew uninstall colima
brew autoremove
sudo rm -rf .lima .colima /etc/sudoers.d/colima /opt/colima
brew install --HEAD colima
colima start --cpu 4 --memory 12 --disk 100
colima list

As well as tried re-installing it using MacPorts, instead of brew:

sudo port install colima

But nothing worked, so far. Any suggestions?

Version

Colima Version: 0.5.5 Lima Version: 0.17.2 Qemu Version: 8.1.0

Operating System

Output of colima status

FATA[0000] colima is not running

Reproduction Steps

  1. brew uninstall colima
  2. brew install colima
  3. colima start

Expected behaviour

colima should start properly.

Additional context

N/A

rmcveigh commented 1 year ago

I am experiencing this issue as well. Only I didn't need to uninstall colima. I was able to reproduce this by doing the following:

  1. colima delete
  2. Reboot my Mac
  3. colima start
nosagadu commented 1 year ago

This error es due to QEMU library that was updated to 8.1.0. You must downgrade to 8.0.4. Link issue lima.

rmcveigh commented 1 year ago

Using the downgrade workaround resolved the issue for me on a MacOS Intel i9. Thanks for the heads up @nosagadu!

AkihiroSuda commented 1 year ago

Duplicate of:

brew uninstall qemu
curl -OSL https://raw.githubusercontent.com/Homebrew/homebrew-core/dc0669eca9479e9eeb495397ba3a7480aaa45c2e/Formula/qemu.rb
brew install ./qemu.rb

Option 2: Install QEMU from the source

brew uninstall qemu
brew install --build-from-source qemu

Option 3: Sign the QEMU binary locally

Lima v0.17.2 shows a prompt to suggest applying this workaround.

cat >entitlements.xml <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>com.apple.security.hypervisor</key>
    <true/>
</dict>
</plist>
EOF

codesign --sign - --entitlements entitlements.xml --force /usr/local/bin/qemu-system-$(uname -m | sed -e s/arm64/aarch64/)
AkihiroSuda commented 1 year ago

The proper fix is being discussed here:

AkihiroSuda commented 1 year ago

Fixed in the qemu bottle 8.1.0-1 🎉

mikaalanwar commented 1 year ago

Duplicate of:

Workarounds

Option 1: Downgrade QEMU to v8.0.3

brew uninstall qemu
curl -OSL https://raw.githubusercontent.com/Homebrew/homebrew-core/dc0669eca9479e9eeb495397ba3a7480aaa45c2e/Formula/qemu.rb
brew install ./qemu.rb

Option 2: Install QEMU from the source

brew uninstall qemu
brew install --build-from-source qemu

Option 3: Sign the QEMU binary locally

Lima v0.17.2 shows a prompt to suggest applying this workaround.

cat >entitlements.xml <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>com.apple.security.hypervisor</key>
    <true/>
</dict>
</plist>
EOF

codesign --sign - --entitlements entitlements.xml --force /usr/local/bin/qemu-system-$(uname -m | sed -e s/arm64/aarch64/)

@AkihiroSuda Thanks a lot! Workaround option # 1 fixed it for me. I'll close this to prevent duplication.