canonical / multipass

Multipass orchestrates virtual Ubuntu instances
https://multipass.run
GNU General Public License v3.0
7.63k stars 635 forks source link

launch failed: timed out waiting for response #3402

Closed stephendoren closed 6 months ago

stephendoren commented 6 months ago

Describe the bug multipass cannot start anymore error shows: "timed out waiting for response"

To Reproduce How, and what happened?

  1. I tried to install multipass with .pkg file and brew method, both failed
  2. It worked several weeks ago. Someday it was not working all of a sudden.

Logs

# multipass launch -vvv
[2024-02-09T09:45:19.626] [debug] [qemu-system-aarch64] [2131] started: qemu-system-aarch64 --version
[2024-02-09T09:45:19.773] [debug] [qemu-img] [2132] started: qemu-img info /var/root/Library/Caches/multipassd/qemu/vault/images/jammy-20240207/ubuntu-22.04-server-cloudimg-arm64.img
[2024-02-09T09:45:19.785] [debug] [qemu-img] [2133] started: qemu-img resize /var/root/Library/Application Support/multipassd/qemu/vault/instances/amiable-potoroo/ubuntu-22.04-server-cloudimg-arm64.img 5368709120
[2024-02-09T09:45:19.807] [debug] [qemu-img] [2134] started: qemu-img snapshot -l /var/root/Library/Application Support/multipassd/qemu/vault/instances/amiable-potoroo/ubuntu-22.04-server-cloudimg-arm64.img
[2024-02-09T09:45:19.816] [debug] [qemu-img] [2135] started: qemu-img amend -o compat=1.1 /var/root/Library/Application Support/multipassd/qemu/vault/instances/amiable-potoroo/ubuntu-22.04-server-cloudimg-arm64.img
[2024-02-09T09:45:19.823] [debug] [amiable-potoroo] process working dir ''
[2024-02-09T09:45:19.823] [info] [amiable-potoroo] process program 'qemu-system-aarch64'
[2024-02-09T09:45:19.823] [info] [amiable-potoroo] process arguments '-machine, virt,gic-version=3, -accel, hvf, -drive, file=/Library/Application Support/com.canonical.multipass/bin/../Resources/qemu/edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on, -cpu, host, -nic, vmnet-shared,model=virtio-net-pci,mac=52:54:00:3e:3a:33, -device, virtio-scsi-pci,id=scsi0, -drive, file=/var/root/Library/Application Support/multipassd/qemu/vault/instances/amiable-potoroo/ubuntu-22.04-server-cloudimg-arm64.img,if=none,format=qcow2,discard=unmap,id=hda, -device, scsi-hd,drive=hda,bus=scsi0.0, -smp, 1, -m, 1024M, -qmp, stdio, -chardev, null,id=char0, -serial, chardev:char0, -nographic, -cdrom, /var/root/Library/Application Support/multipassd/qemu/vault/instances/amiable-potoroo/cloud-init-config.iso'
[2024-02-09T09:45:19.825] [debug] [qemu-system-aarch64] [2136] started: qemu-system-aarch64 -machine virt,gic-version=3 -nographic -dump-vmstate /private/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/multipassd.bBhTbV
[2024-02-09T09:45:19.863] [info] [amiable-potoroo] process state changed to Starting
[2024-02-09T09:45:19.866] [info] [amiable-potoroo] process state changed to Running
[2024-02-09T09:45:19.866] [debug] [qemu-system-aarch64] [2137] started: qemu-system-aarch64 -machine virt,gic-version=3 -accel hvf -drive file=/Library/Application Support/com.canonical.multipass/bin/../Resources/qemu/edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on -cpu host -nic vmnet-shared,model=virtio-net-pci,mac=52:54:00:3e:3a:33 -device virtio-scsi-pci,id=scsi0 -drive file=/var/root/Library/Application Support/multipassd/qemu/vault/instances/amiable-potoroo/ubuntu-22.04-server-cloudimg-arm64.img,if=none,format=qcow2,discard=unmap,id=hda -device scsi-hd,drive=hda,bus=scsi0.0 -smp 1 -m 1024M -qmp stdio -chardev null,id=char0 -serial chardev:char0 -nographic -cdrom /var/root/Library/Application Support/multipassd/qemu/vault/instances/amiable-potoroo/cloud-init-config.iso
[2024-02-09T09:45:19.866] [info] [amiable-potoroo] process started
launch failed: The following errors occurred:
amiable-potoroo: timed out waiting for response

Additional info

townsend2010 commented 6 months ago

Hi @stephendoren!

Sorry you are having this issue. Could you please provide more logs in order for us to better help diagnose the issue? Thanks!

akauppi commented 6 months ago

The 14.3.1 (not 14.3) may be a key, here.

I updated macOS today, and started getting something similar on one VM that was in "running" state over the update, and had one "native" mount as well. Did not have other VM's so cannot say do much compares. Creating a new VM works.

First, after the update:

$ multipass shell rust
shell failed: qemu-system-x86_64: -accel hvf: Unknown Error

This is also listed as a warning (not an error), in the /Library/Logs/Multipass/multipassd.log:

[2024-02-09T15:48:29.119] [info] [rust] process started
[2024-02-09T15:48:29.126] [debug] [rust] Waiting for SSH to be up
[2024-02-09T15:48:29.131] [info] [daemon] Starting Multipass 1.13.0+mac
[2024-02-09T15:48:29.131] [info] [daemon] Daemon arguments: /Library/Application Support/com.canonical.multipass/bin/multipassd --verbosity debug
[2024-02-09T15:48:29.147] [warning] [rust] qemu-system-x86_64: -accel hvf: Unknown Error

[2024-02-09T15:48:29.147] [warning] [qemu-system-x86_64] 
[2024-02-09T15:48:29.147] [info] [rust] process state changed to NotRunning
[2024-02-09T15:48:29.147] [error] [rust] process error occurred Crashed program: qemu-system-x86_64; error: Process crashed
[2024-02-09T15:48:29.148] [error] [rust] error: program: qemu-system-x86_64; error: Process crashed

I restarted the VM, thinking that might help. After the restart:

$ mp info
info failed: ssh connection failed: 'Timeout connecting to 192.168.64.74'

Edit:

Got the original VM up, by (macOS side):

$ sudo launchctl unload /Library/LaunchDaemons/com.canonical.multipassd.plist
$ sudo launchctl load /Library/LaunchDaemons/com.canonical.multipassd.plist

Also remembered: within the VM, the kernel version had recently changed from 5.15.0-92 to 5.15.0-94, by apt upgrade. I needed to update one driver because of this - no idea if these are connected. Hoping to be helpful; sorry if noise.

stephendoren commented 6 months ago

I uninstalled and re-installed multipass, now it keeps asking me to authenticate the service, as below:

launch failed: The client is not authenticated with the Multipass service.
Please use 'multipass authenticate' before proceeding.

I followed the instruction from this page: https://multipass.run/docs/authenticating-clients, but not working. It seems to fall into a bad loop.

# multipass launch -vvv         
launch failed: The client is not authenticated with the Multipass service.
Please use 'multipass authenticate' before proceeding.

# multipass authenticate
Please enter passphrase: 
authenticate failed: Passphrase is not set. Please `multipass set local.passphrase` with a trusted client.

# multipass set local.passphrase
Please enter passphrase: 
Please re-enter passphrase: 
set failed: The client is not authenticated with the Multipass service.
Please use 'multipass authenticate' before proceeding.
townsend2010 commented 6 months ago

Hi @stephendoren,

It appears a different user authenticated first with the daemon. Did you happen to use sudo multipass .. after (re)installing Multipass? You could try to see if something like sudo multipass list works. If this does, then you can do $ multipass set local.passphrase and set a passphrase that you can then use to authenticate with your normal user.

If this doesn't work, then you could try these steps under the "How to recover on macOS" section.

stephendoren commented 6 months ago

Hi @townsend2010,

Thanks a lot. The problem was resolved by this:

If this doesn't work, then you could try these steps under the "How to recover on macOS" section.

Now my Multipass is working.

townsend2010 commented 6 months ago

Hi @stephendoren!

Glad that got it working for you! Are saying that the original issue you reported here is no longer occurring and we can close this or is there still an issue? Thanks!

stephendoren commented 6 months ago

Are saying that the original issue you reported here is no longer occurring and we can close this or is there still an issue? Thanks!

Yea, issue closed. Thanks.

townsend2010 commented 6 months ago

Thanks!