search

canonical / multipass

Multipass orchestrates virtual Ubuntu instances
https://multipass.run
GNU General Public License v3.0
7.95k stars 651 forks source link

[regression][windows] Changing `MULTIPASS_STORAGE` makes daemon inaccessible #3184

Closed andrei-toterman closed 1 year ago

andrei-toterman commented 1 year ago

Describe the bug Changing the MULTIPASS_STORAGE location on Windows results in the daemon being inaccessible by the client.

To Reproduce How, and what happened?

  1. Stop-Service Multipass
  2. Set-ItemProperty -Path "HKLM:System\CurrentControlSet\Control\Session Manager\Environment" -Name MULTIPASS_STORAGE -Value "C:\Users\andrei\mp"
  3. Copy data from C:\ProgramData\Multipass to the MULTIPASS_STORAGE directory
  4. Start-Service Multipass
  5. multipass daemon fails to authenticate the client

Expected behavior Multipass to work the same no matter what MULTIPASS_STORAGE value is set.

Additional info

Additional context Issue first described here: https://discourse.ubuntu.com/t/multipass-issues-after-upgrade-to-multipass-1-12-2-win-win64-on-windows-11/37170

kohtala commented 1 year ago

I upgraded from 1.11.1+win to 1.12.2+win with the MULTIPASS_STORAGE set. The multipass CLI reports The client is not authenticated with the Multipass service.

Also trying to start the virtual machine from HYPER-V, the HYPER-V reports access denied 0x80070005 error for the data\vault\instances vhdx files.

The link to discourse seems to describe same problems.

sharder996 commented 1 year ago

Hi @kohtala, thanks for the extra info. I think I know what's going on here and a fix should be coming soon!

rageshkrishna commented 1 year ago

I just ran into this myself. I had no real reason to upgrade to 1.12.2, but decided to do so anyway to stay up to date. At first I thought it had "forgotten" about my MULTIPASS_STORAGE settings so the first hour or so I was chasing that down and wondering why it had no effect. Then I came across this issue and the linked discourse thread.

For the benefit of anyone else that is trying to do this, here's what worked for me to downgrade to 1.11.1:

This appears to have gotten me back to a working state with 1.11.1

@sharder996 I'm looking forward to 1.12.3. Do you happen to have an ETA for that release?

sharder996 commented 1 year ago

Hi @rageshkrishna,

Yes, this has been looked at and fixed in the current development branch. Since we haven't been hearing much about this issue from our Windows users we had decided to instead release the fix in 1.13 which will be coming out in just over a month.

However, in the meantime, if you're still having issues with this, we have updated documentation to make the setting up of MULTIPASS_STORAGE more reliable as well as a temporary workaround for users encountering issues with disk resizing because of Windows permissions. See #3216.

Hopefully this helps, but let me know if you are still having issues!

rageshkrishna commented 1 year ago

Thanks for the update @sharder996. I'm currently in no hurry to upgrade from 1.11 as it gives me everything I need.

I think it's unfortunate that this doesn't warrant a 1.12.x patch because it is quite a bad bug to have, especially considering MULTIPASS_STORAGE is a documented and supported way of running things. That being said, I totally understand release priorities. I'm looking forward to trying out 1.13 when it gets released.

kohtala commented 10 months ago

I noticed 1.13.0+win was available and upgraded hoping to get back to the VM. Already during install there were errors since multicloud service did not start.

Log shows (with the VM name hidden for privacy) [**] [15072] Cmdlet: 'Get-VMCheckpoint -VMName ** | Where-Object { $_.IsAutomaticCheckpoint } | Remove-VMCheckpoint -Confirm:$false' followed by [**] [15072] Cmdlet exit status is 'false'

The permissions of the directories were not fixed on upgrade. I found another VM and used the PowerShell as administrator Get-Acl <NEW_STORAGE_DIR> | Set-Acl <OLD_STORAGE_DIR>.old trick to copy ACL on the directory pointed by MULTIPASS_STORAGE as well as for the data\vault\instances\** directory and files in it. After that the multipass service stayed alive. For HYPER-V to be able to start the VM, parent directories also needed some additional permissions.

I also tried the steps in 3216 comment. Restarted multipass service.

The login failure still remained as it also was not fixed on upgrade. There is documentation, but it only documents for Linux.

I found the client certificate multipass_cert.pem (three of them, actually) at %APPDATA%\multipass\client-certificate, %APPDATA%\multipass-gui\client-certificate and %LOCALAPPDATA%\multipass-client-certificate. The one in %LOCALAPPDATA% was in %MULTIPASS_STORAGE%\data\authenticated-certs\multipass_client_certs.pem, but the multipass_client_certs.pem ACL apparently was not sufficient for the multipass service.

Once I got ACL copied from an original location to multipass_client_certs.pem and restarted multipass, multipass CLI was able to authenticate.

sharder996 commented 10 months ago

Hi @kohtala!

I tried playing around with MULTIPASS_STORAGE and transitioning between different versions, but I haven't been able to reproduce what you're describing. If you are still having issues and this is a problem for you, could you open up a new github issue so that we can properly track it?

Thanks!

kohtala commented 10 months ago

I had the problem as described at the discourse link. I was waiting for the upgrade in hopes it would fix it.

Since 1.13.0+win upgrade did not fix up the mess caused by upgrade to 1.12.2+win with MULTIPASS_STORAGE set, I had to figure out how to get it working. I was just documenting the problems I found and what I did to get it to somewhat working condition. In case it helps anyone else.

It's still not restored fully. Hyper-V and multipass are not in sync with the starting and state of the virtual machine. But at least I can start it from HYPER-V and check if I had anything to salvage in the VM.