flying-sausages commented 3 years ago

What are you trying to do? Run Multipass to launch instances on top of Apple Silicon devices

What's your proposed solution? Restrict multipass on AS to use the Virtualisation API intrdouced in Big Sur only and ensure ARM builds of ubuntu are being installed

Additional context I have 0 experience in any of the underlying technologies that are used here. I know 1.5.0 has Big Sur support on Intel through the API above, and so I'm wondering what are the extra steps that need to be taken care in order to allow users to have ARM-based VMs.

ED: Latest build

1.8.0 build 787

Saviq commented 3 years ago

Hi @flying-sausages, this is definitely something we want to get going, but it will take some time. The API you mention is quite low level, the likes of hyperkit and qemu are what we'd need to implement support for it first. Did you see evidence that the hardware even supports virtualization already? From what we've read the dev kits did not, and I've not seen anyone confirming they have a VM running under Apple Silicon.

flying-sausages commented 3 years ago

I've seen a success story here which I think could be helpful https://forums.macrumors.com/threads/ubuntu-linux-virtualized-on-m1-success.2270365/

flying-sausages commented 3 years ago

Another useful link from the same thread https://github.com/JacopoMangiavacchi/M1-Linux-SSH

flying-sausages commented 3 years ago

And another https://github.com/evansm7/vftool

RyanMorash commented 3 years ago

The M1 does have virtualization support. There have been a set of patches made to enable Hypervisor.framework (hvf) support to QEMU for aarch64: https://patchwork.kernel.org/project/qemu-devel/list/?series=391797

Also, there are now two different virtualization frameworks on macOS. There continues to be the lower-level Hypervisor.framework which is currently utilized by hyperkit via machyve/xhyve. Virtualization.framework is a new higher-level set of APIs which allows you to create VMs using the Virtio spec without having to deal with a separate hypervisor (such as xhyve)

Saviq commented 3 years ago

Yeah thanks @RyanMorash @flying-sausages for references, we're monitoring this space and will jump on it whenever things settle. One notorious source of problems for us is networking (NAT, bridging, DHCP, DNS), which hyperkit (through Hypervisor.framework) mostly solves and we'd love to continue using that. If, however, qemu gains support for all the bits'n'pieces we need, we may well go that route.

flying-sausages commented 3 years ago

https://gist.github.com/niw/e4313b9c14e968764a52375da41b4278#file-readme-md looks like there's QEMU support through some patches? Lively discussion happening underneath that.

RyanMorash commented 3 years ago

Docker is moving away from using HyperKit (which they originally created for Docker Desktop) and switching to Virtualization.framework: https://www.docker.com/blog/download-and-try-the-tech-preview-of-docker-desktop-for-m1/

ChangheeOh commented 3 years ago

I have tried to create VM with vmcli(https://github.com/gyf304/vmcli) and almost succeeded to run Ubuntu arm64 image. vmcli is also based on MacOS Virtualization.framework and it also have several issues and not yet stable in my opinion. I wish it could help multipass team to enhance it to support M1 Mac.

nottrobin commented 3 years ago

It seems docker have made good progress: https://docs.docker.com/docker-for-mac/apple-m1/

Saviq commented 3 years ago

@nottrobin yeah we know everything we need now, just need to roadmap it - hoping for early next cycle.

andy-isoc commented 3 years ago

Is there a good place (other than this issue!) to watch for updates on this and when it might be released?

bassu commented 3 years ago

There is also ACVM (https://github.com/KhaosT/ACVM) GUI qemu launcher (based on patches from Alexander Graf to support Hypervisor.framework syscalls on M1 macOS).

But I suppose if multipass uses Hyperkit then it might take some time!

sudo8com commented 3 years ago

The complete guide for MacOS Apps Optimized for M1 Apple Silicon Macs https://isapplesiliconready.com/

flying-sausages commented 3 years ago

This open-source implementation works quite well, wonder if it can be of any inspiration. https://github.com/utmapp/UTM

misterproton commented 3 years ago

@Saviq can you update us on this issue? No new Intel Macs will be designed, so us Mac Multipass users are kind of in the cold here.

Saviq commented 3 years ago

Hey @misterproton, all, we've scheduled the work on M1 support for the development cycle that just started. We'll be starting the work on it within a couple weeks. Stay tuned!

misterproton commented 3 years ago

@Saviq do you have any update or ETA for this one?

Saviq commented 3 years ago

Hi @misterproton we're actively working on it now. We should have a first build by July.

rugwirobaker commented 3 years ago

:eyes:

flying-sausages commented 3 years ago

Happy to hear! Is there a chance this will show up in the 1.7.0 RCs?

townsend2010 commented 3 years ago

Hey @flying-sausages!

No, it definitely won't be in the 1.7.0 release, but the plan is to make some beta packages available once I get this in a relatively good working state, most likely some time in July. I'm actively working on it :smiley:

rugwirobaker commented 3 years ago

How is going so far, are we still on track(no pressure😉)?

townsend2010 commented 3 years ago

@rugwirobaker, still working on it :wink:

ghost commented 3 years ago

Big thanks for the work on this Chris. @townsend2010

b4s36t4 commented 3 years ago

Can we expect the build by end of July or it's gonna take more time? Waiting is hard. 🥲

townsend2010 commented 3 years ago

Hey @b4s36t4, et. al.,

Yeah, I'll give an update on where I am on this.

I've made some really good progress, but it looks quite unlikely we'll have a beta in time for the end of July. The good news is that I've been able to launch Multipass instances on an M1. The bad news is that M1-based Macs are very strict on signing of blobs inside a package (like libraries we pull in), so trying to get a full beta package that is not "officially" signed by Canonical (like our released packages), but still run on anybody's M1 is proving to be difficult. Yay Apple and their packaging :wink:

One other hurdle I'm trying to overcome is the ability to launch more than one or two instances at a time is problematic. I have an idea on how to fix it, but it will take a little time.

That said, I'm aiming for August to get a beta out as long as we can distribute an unofficially signed package that will run on your M1.

Thanks and stay tuned!

ben-ihelputech commented 3 years ago

Not saying that I would be able to help in any specific way, but is there anything the community can do/research to help you all out? Thanks for the updates! We're all excited to see Multipass support for the M1's

townsend2010 commented 3 years ago

Hi @ben-ihelputech!

When we do have a beta, that is when the community can help- ensuring that what we have is good! In the meantime, I appreciate the offer, but I have a handle on what needs to be done and just trying to manage the time between other job duties, summer holidays, etc. :smiley:

I'm excited to see Multipass support for M1 too!

ben-ihelputech commented 3 years ago

Awesome! Don't work yourselves too hard hihi.

Saviq commented 3 years ago

Hi all!

Courtesy of @townsend2010 we have a first package that you can try out:

https://multipass-ci.s3.amazonaws.com/pr403/multipass-1.8.0-dev.787.pr403%2Bg9c2fa6ea.mac%2Barm64-Darwin.pkg

And it's pretty quick, too:

# having launched before, so the image is cached
$ time multipass launch
Launched: productive-cattle
multipass launch  0,12s user 0,02s system 0% cpu 25,110 total

I actually saw it launch below 20s.

Please give it a run and let us know how it goes!

jamal commented 3 years ago

Amazing, this works great! Thank you so much.

❯ multipass list
Name                    State             IPv4             Image
queenly-blackbird       Running           192.168.64.3     Ubuntu 20.04 LTS

offby1 commented 3 years ago

I downloaded it, installed it, and tried to run it; but all it did was very briefly show an animated expanding icon. No app window got created. I assume that means it crashed at startup. I've attached a crash log.

On Tue, Aug 10, 2021 at 9:55 AM Michał Sawicz @.***> wrote:

Hi all!

Courtesy of @townsend2010 https://github.com/townsend2010 we have a first package that you can try out:

https://multipass-ci.s3.amazonaws.com/multipass-1.8.0-dev.397%2Bg9ef3f5c4.mac%2Barm64-Darwin.pkg

And it's pretty quick, too:

having launched before, so the image is cached

$ time multipass launch Launched: productive-cattle multipass launch 0,12s user 0,02s system 0% cpu 25,110 total

I actually saw it launch below 20s.

Please give it a run and let us know how it goes!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/canonical/multipass/issues/1857#issuecomment-896149705, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAYSLG23HSGBKS3CUH43TT4FKZZANCNFSM4UCTTM7A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

kristaxox commented 3 years ago

Worked great for me, although I also noticed that the menubar and app windows did not launch.

kristaxox$ multipass launch groovy
...
Launched: cute-trout

kristaxox$ multipass shell cute-trout
Welcome to Ubuntu 20.10 (GNU/Linux 5.8.0-63-generic aarch64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Tue Aug 10 13:17:36 EDT 2021

  System load:  0.49              Processes:               93
  Usage of /:   26.3% of 4.68GB   Users logged in:         0
  Memory usage: 18%               IPv4 address for enp0s2: 192.168.64.3
  Swap usage:   0%

0 updates can be installed immediately.
0 of these updates are security updates.

The list of available updates is more than a week old.
To check for new updates run: sudo apt update
Your Ubuntu release is not supported anymore.
For upgrade information, please visit:
http://www.ubuntu.com/releaseendoflife

New release '21.04' available.
Run 'do-release-upgrade' to upgrade to it.

To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

ubuntu@cute-trout:~$

OS: macOS 11.2 20D64 arm64
Host: Macmini9,1
Kernel: 20.3.0

offby1 commented 3 years ago

D'oh! I didn't think to try it from the command line. "multipass shell" works fine :-)

On Tue, Aug 10, 2021 at 10:37 AM Krista @.***> wrote:

Worked great for me, although I also noticed that the menubar and app windows did not launch.

kristaxox$ multipass launch groovy ... Launched: cute-trout

kristaxox$ multipass shell cute-trout Welcome to Ubuntu 20.10 (GNU/Linux 5.8.0-63-generic aarch64)

Documentation: https://help.ubuntu.com

Management: https://landscape.canonical.com

Support: https://ubuntu.com/advantage

System information as of Tue Aug 10 13:17:36 EDT 2021

System load: 0.49 Processes: 93 Usage of /: 26.3% of 4.68GB Users logged in: 0 Memory usage: 18% IPv4 address for enp0s2: 192.168.64.3 Swap usage: 0%

0 updates can be installed immediately. 0 of these updates are security updates.

The list of available updates is more than a week old. To check for new updates run: sudo apt update Your Ubuntu release is not supported anymore. For upgrade information, please visit:http://www.ubuntu.com/releaseendoflife

New release '21.04' available. Run 'do-release-upgrade' to upgrade to it.

To run a command as administrator (user "root"), use "sudo ". See "man sudo_root" for details.

@.***:~$

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/canonical/multipass/issues/1857#issuecomment-896181584, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAYSNUQ3QDBBXY2NTFPYLT4FPTZANCNFSM4UCTTM7A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

Saviq commented 3 years ago

@offby1 @kristaxox yeah we're primarily CLI based, but there is a small GUI component that goes in the menu bar, but indeed that's broken currently. Again the signature requirements of M1 macOS are biting us :)

cafsenra commented 3 years ago

Doesn't work for me:

➜  ~ ls -la /var/run/multipass_socket
ls: /var/run/multipass_socket: No such file or directory
➜  ~ multipass info --all
info failed: cannot connect to the multipass socket
Please ensure multipassd is running and '/var/run/multipass_socket' is accessible

Apparently she installer doesn't install Multipass_socket.

townsend2010 commented 3 years ago

Hi @cafsenra,

The installer doesn't create the socket. The daemon creates that on start up and it looks like the daemon is not running. Could you please post the log found at /Library/Logs/Multipass/multipassd.log?

cafsenra commented 3 years ago

Hi @townsend2010,

I already managed to run Multipass. Completely uninstall and clean Multipass, reinstall it and now works! ;)

before:

➜  ~ sudo cat /Library/Logs/Multipass/multipassd.log
[2021-08-10T19:51:21.089] [error] [daemon] Caught an unhandled exception: Unsupported virtualization driver: hyperkit
[2021-08-10T19:51:31.233] [error] [daemon] Caught an unhandled exception: Unsupported virtualization driver: hyperkit
[2021-08-10T19:51:41.380] [error] [daemon] Caught an unhandled exception: Unsupported virtualization driver: hyperkit

after:

[2021-08-10T20:08:21.148] [debug] [ssh session] Executing 'sudo /bin/bash -c 'cd "/home/ubuntu/" && mkdir -p "Home"''
[2021-08-10T20:08:21.152] [debug] [ssh process] /Users/michal/dev/multipass-private/src/ssh/ssh_process.cpp:118 read_stream(type = 0, timeout = -1):
[2021-08-10T20:08:21.152] [debug] [ssh process] /Users/michal/dev/multipass-private/src/ssh/ssh_process.cpp:124 read_stream(): channel closed
[2021-08-10T20:08:21.152] [debug] [ssh process] /Users/michal/dev/multipass-private/src/ssh/ssh_process.cpp:118 read_stream(type = 1, timeout = -1):
[2021-08-10T20:08:21.152] [debug] [ssh process] /Users/michal/dev/multipass-private/src/ssh/ssh_process.cpp:124 read_stream(): channel closed
[2021-08-10T20:08:21.152] [debug] [ssh session] Executing 'sudo /bin/bash -c 'cd "/home/ubuntu/" && chown -R 1000:1000 "Home"''
[2021-08-10T20:08:21.157] [debug] [ssh process] /Users/michal/dev/multipass-private/src/ssh/ssh_process.cpp:118 read_stream(type = 0, timeout = -1):
[2021-08-10T20:08:21.157] [debug] [ssh process] /Users/michal/dev/multipass-private/src/ssh/ssh_process.cpp:124 read_stream(): channel closed
[2021-08-10T20:08:21.157] [debug] [ssh process] /Users/michal/dev/multipass-private/src/ssh/ssh_process.cpp:118 read_stream(type = 1, timeout = -1):
[2021-08-10T20:08:21.157] [debug] [ssh process] /Users/michal/dev/multipass-private/src/ssh/ssh_process.cpp:124 read_stream(): channel closed

I don't know who's Michal, but it is a weird log output!

townsend2010 commented 3 years ago

Hi @cafsenra,

Yeah, it looks like you had the non-compatible version of Multipass previously installed and completely removing it cleared up the cruft.

lol, yeah, this beta was built on a local machine, hence the, uh, unique path :wink: Glad you got it working!

ghost commented 3 years ago

Tested on MacBook Air (M1, 2020). Looks good as below.

Let me know if you need more specific tests.

MacBook-Air-M1:~ mrsnakeoil$ multipass launch
One quick question before we launch … Would you like to help
the Multipass developers, by sending anonymous usage data?
This includes your operating system, which images you use,
the number of instances, their properties and how long you use them.
We’d also like to measure Multipass’s speed.
Send usage data (yes/no/Later)? yes
Thank you!
Launched: purified-cicada
MacBook-Air-M1:~ mrsnakeoil$

flying-sausages commented 3 years ago

Would be sweet to have #1509 looked at as well seeing as zsh is the default shell on macos for a while but no biggie

shinokada commented 3 years ago

Great! It works better than the x86_64 version for me. Before I always had to run echo "nameserver 8.8.8.8" | sudo tee /etc/resolv.conf > /dev/null before running the sudo apt update command. Now it works like a charm.

One thing to mention is that I didn't see any progress bars after multipass start command. I got a hundred of blank lines as the following image.

lcmartinezdev commented 3 years ago

Works properly. Macbook Pro M1, 16gb RAM

multipass start

Launched: primary
Mounted '/Users/USERNAME' into 'primary:Home'

multipass info primary

Name:           primary
State:          Running
IPv4:           192.168.64.2
Release:        Ubuntu 20.04.2 LTS
Image hash:     42704930377d (Ubuntu 20.04 LTS)
Load:           0.20 0.09 0.03
Disk usage:     1.2G out of 4.7G
Memory usage:   151.6M out of 974.8M
Mounts:         /Users/USERNAME => Home
                    UID map: 501:default
                    GID map: 20:default

shawnanastasio commented 3 years ago

Excited by the news, I cloned the multipass repo and checked out the macos-add-qemu-support branch but to my surprise it seems the build system is entirely unprepared to handle macOS at all. Am I correct in assuming that this repository only contains a subset of the multipass code and that the macOS support is only present in the proprietary builds by Canonical?

Saviq commented 3 years ago