[macos] instance unknown state / cannot be started

DominicBurkart commented 1 year ago

Describe the bug

All instances are in an unknown state after unexpected restart due to battery issue, and new instances cannot be created.

I assume that I need to delete some qemu images / configurations, but am not sure how to proceed. If possible, it would be awesome if multipass could recover from this issue!

To Reproduce How, and what happened?

Multipass works as expected (e.g. running multipass launch --cpus='5' --memory='8G' --disk='64g' --name ad && multipass shell ad).
Then, the computer runs out of battery while VM is running.
When device is booted, the instance is in state Unknown and cannot run. The instance can be stopped (or directly deleted and purged), but cannot be started or a shell opened. New instances can be created but not started or a shell opened.
Issue is not resolved by uninstalling/reinstalling multipass using the installer or using brew.

Expected behavior

multipass can identify and recover the issue (underlying driver misconfiguration?), allowing new instances to be created.

Logs Please provide logs from the daemon, see accessing logs on where to find them on your platform.

From running multipass launch --cpus='5' --memory='8G' --disk='64g' --name ad && multipass shell ad:

[2023-09-06T10:11:04.799] [debug] [qemu-system-aarch64] [42171] started: qemu-system-aarch64 --version
[2023-09-06T10:11:05.074] [debug] [qemu-img] [42172] started: qemu-img info /var/root/Library/Caches/multipassd/qemu/vault/images/jammy-20230828/ubuntu-22.04-server-cloudimg-arm64.img
[2023-09-06T10:11:05.083] [debug] [qemu-img] [42173] started: qemu-img resize /var/root/Library/Application Support/multipassd/qemu/vault/instances/ad/ubuntu-22.04-server-cloudimg-arm64.img 68719476736
[2023-09-06T10:11:05.090] [debug] [qemu-img] [42174] started: qemu-img snapshot -l /var/root/Library/Application Support/multipassd/qemu/vault/instances/ad/ubuntu-22.04-server-cloudimg-arm64.img
[2023-09-06T10:11:05.095] [debug] [ad] process working dir ''
[2023-09-06T10:11:05.095] [info] [ad] process program 'qemu-system-aarch64'
[2023-09-06T10:11:05.095] [info] [ad] process arguments '-machine, virt,gic-version=3, -accel, hvf, -drive, file=/Library/Application Support/com.canonical.multipass/bin/../Resources/qemu/edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on, -cpu, host, -nic, vmnet-shared,model=virtio-net-pci,mac=52:54:00:8c:64:6f, -device, virtio-scsi-pci,id=scsi0, -drive, file=/var/root/Library/Application Support/multipassd/qemu/vault/instances/ad/ubuntu-22.04-server-cloudimg-arm64.img,if=none,format=qcow2,discard=unmap,id=hda, -device, scsi-hd,drive=hda,bus=scsi0.0, -smp, 5, -m, 8192M, -qmp, stdio, -chardev, null,id=char0, -serial, chardev:char0, -nographic, -cdrom, /var/root/Library/Application Support/multipassd/qemu/vault/instances/ad/cloud-init-config.iso'
[2023-09-06T10:11:05.098] [debug] [qemu-system-aarch64] [42175] started: qemu-system-aarch64 -machine virt,gic-version=3 -nographic -dump-vmstate /private/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/T/multipassd.zAFszV
[2023-09-06T10:11:05.339] [info] [ad] process state changed to Starting
[2023-09-06T10:11:05.342] [info] [ad] process state changed to Running
[2023-09-06T10:11:05.342] [debug] [qemu-system-aarch64] [42178] started: qemu-system-aarch64 -machine virt,gic-version=3 -accel hvf -drive file=/Library/Application Support/com.canonical.multipass/bin/../Resources/qemu/edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on -cpu host -nic vmnet-shared,model=virtio-net-pci,mac=52:54:00:8c:64:6f -device virtio-scsi-pci,id=scsi0 -drive file=/var/root/Library/Application Support/multipassd/qemu/vault/instances/ad/ubuntu-22.04-server-cloudimg-arm64.img,if=none,format=qcow2,discard=unmap,id=hda -device scsi-hd,drive=hda,bus=scsi0.0 -smp 5 -m 8192M -qmp stdio -chardev null,id=char0 -serial chardev:char0 -nographic -cdrom /var/root/Library/Application Support/multipassd/qemu/vault/instances/ad/cloud-init-config.iso
[2023-09-06T10:11:05.342] [info] [ad] process started
[2023-09-06T10:11:05.343] [debug] [ad] Waiting for SSH to be up
[2023-09-06T10:11:05.734] [debug] [ad] QMP: {"QMP": {"version": {"qemu": {"micro": 0, "minor": 0, "major": 8}, "package": ""}, "capabilities": ["oob"]}}

[2023-09-06T10:11:05.795] [debug] [ad] QMP: {"return": {}}

[2023-09-06T10:11:19.068] [debug] [ad] QMP: {"timestamp": {"seconds": 1693987879, "microseconds": 68934}, "event": "NIC_RX_FILTER_CHANGED", "data": {"path": "/machine/unattached/device[10]/virtio-backend"}}

Additional info

OS: Ventura 13.4.1

multipass version

multipass   1.12.2+mac
multipassd  1.12.2+mac

multipass info --all

Name:           ad
State:          Unknown
IPv4:           --
Release:        --
Image hash:     41a8587b4994 (Ubuntu 22.04 LTS)
CPU(s):         --
Load:           --
Disk usage:     --
Memory usage:   --
Mounts:         --

multipass get local.driver
```
qemu
```

Additional context

In my case, the instance does not need to be recovered; priority is just being able to create new instances.

Similar issues identified:

I don't think any of the solutions there are relevant to this case, where a full reinstall didn't solve the issue.

DominicBurkart commented 1 year ago

Running brew reinstall qemu then brew uninstall --zap --cask multipass && brew install multipass didn't resolve the issue.

DominicBurkart commented 1 year ago

Killing all processes associated with multipass (as recommended here: https://gist.github.com/zulhfreelancer/12817fa5d1579bb640ed00dff8327a4a) also doesn't resolve the issue.

DominicBurkart commented 1 year ago

resetting the network as described here also did not work: https://multipass.run/docs/troubleshoot-networking#heading--network-routing-problems

miseyu commented 1 year ago

I'm in the same situation.

OS: Ventura 13.5
multipass version
multipass   1.12.2+mac
multipassd  1.12.2+mac

miseyu commented 1 year ago

This one has been resolved. What we did

Update mac os to the latest version
Reinstalled multipass with brew

$ sw_vers
ProductName:            macOS
ProductVersion:         13.5.2
BuildVersion:           22G91

miseyu commented 1 year ago

However, since we don't know the root cause, the phenomenon is likely to occur again, and we would like to continue to watch for it.

kwagga commented 1 year ago

Ran into the same issue and the reinstallation of qemu and multipass via brew did not resolve it for me.

❯ sw_vers
ProductName:        macOS
ProductVersion:     13.5.1
BuildVersion:       22G90

❯ multipass version
multipass   1.12.2+mac
multipassd  1.12.2+mac

andrei-toterman commented 1 year ago

Hey, @DominicBurkart! From the way you described it so far, it sounds like a case of #2387. Is the firewall enabled on your machine? If so, please try to disable it, reboot and try using Multipass again. You can also try running the following commands to allow bootpd to run.

$ sudo su -
$ /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/libexec/bootpd
$ /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd
$ reboot now

But it might not be the same issue as that one, so please let me know if this doesn't work

bitsf commented 1 year ago

I met the same problem just from this month. I tried to run socketfilterfw, but got error Firewall settings cannot be modified from command line on managed Mac computers. Is there any other method to workaround?

georgeliao commented 1 year ago

Hi @bitsf I am afraid that we do not have other workarounds except the one that has been discussed. https://github.com/canonical/multipass/issues/2387 This post has more info, but the socketfilterfw is the distilled information from that post.

bitsf commented 1 year ago

Finally, I added /usr/libexec/bootpd into firewall from GUI, and reboot Mac, then it can work now.

bitsf commented 1 year ago

my multipass can't start again after several days. And I find if I run sudo qemu-system-aarch64 -machine virt,gic-version=3 -accel hvf -drive "file=/Library/Application Support/com.canonical.multipass/bin/../Resources/qemu/edk2-aarch64-code.fd,if=pflash,format=raw,readonly=on" -cpu host -nic vmnet-shared,model=virtio-net-pci,mac=52:54:00:f6:c6:64 -device virtio-scsi-pci,id=scsi0 -drive "file=/var/root/Library/Application Support/multipassd/qemu/vault/instances/primary/ubuntu-22.04-server-cloudimg-arm64.img,if=none,format=qcow2,discard=unmap,id=hda" -device scsi-hd,drive=hda,bus=scsi0.0 -smp 1 -m 1024M -cdrom "/var/root/Library/Application Support/multipassd/qemu/vault/instances/primary/cloud-init-config.iso" -display default,show-cursor=on -qmp stdio, it seems failed to start the VM

But if I remove the gic-version=3, just use -machine virt or use -machine virt,gic-version=2, it can start the VM. I'm not sure if this is an issue

percygrunwald commented 1 year ago

I was able to resolve this issue using the OnyX tool as recommended in another GH issue relating to VM start issues: https://github.com/canonical/multipass/issues/3003#issuecomment-1736353490

vielmetti commented 1 year ago

I ran into exactly this problem. Looking into Onyx

jrz commented 1 year ago

Same here after fresh install Sonoma on M1. Pretty useless software like this. How do I completely remove everything including all the gunk?

yzcyayaya commented 10 months ago

I'm having the same problem, how can I solve it?

Firewall off statu

~$ multipass list
Name                    State             IPv4             Image
repository-host         Unknown           --               Ubuntu 22.04 LTS
kube-master             Unknown           --               Ubuntu 22.04 LTS
kube-node01             Unknown           --               Ubuntu 22.04 LTS
kube-node02             Unknown           --               Ubuntu 22.04 LTS

~$ sw_vers
ProductName:        macOS
ProductVersion:     13.4.1
BuildVersion:       22F82

~$ multipass version
multipass   1.12.2+mac
multipassd  1.12.2+mac

yzcyayaya commented 10 months ago

/cc @townsend2010 Is this situation included in the restoration plan?

townsend2010 commented 10 months ago

Hi @yzcyayaya!

I'm having the same problem, how can I solve it?

Could you please provide the log?

Is this situation included in the restoration plan?

Sorry, I'm not really sure what you mean by the "restoration plan."

yzcyayaya commented 10 months ago

Is this situation included in the restoration plan?

Haha, it's okay, I solved my needs using kvm from another machine

krmao commented 2 days ago

➜  .output git:(pc) multipass exec docker docker
exec failed: Cannot retrieve credentials in unknown state
➜  .output git:(pc) multipass shell docker
shell failed: Cannot retrieve credentials in unknown state
➜  .output git:(pc)

canonical / multipass

[macos] instance unknown state / cannot be started #3220

I'm having the same problem, how can I solve it?