Consider a different location for Multipass files on macOS

[BeauSlim]

I imagine that, like me, most mac users of virtualization choose to exclude VM disk images from Time Machine backups. Multipass on macOS puts its files in /var/root/Library/Application Support/multipassd/. There are problems with this:

1. It is undocumented, and users have to go digging to find out where these images are in order to add them to their TM excluded directory list.

2. You can't get there in the Finder. To exclude multipass' folders, you either have to do some ninja "sudo open" stuff or use the tmutil command line (i.e. "sudo tmutil addexclusion -p /var/root/Library/Application\ Support/multipassd").

3. If a user wants to use TM to backup their multipass images, recovery is extremely awkward since macOS has this location pretty locked down.

I can see that you're trying to keep things really simple and assume this is the reason to keep multipass' disk images out of a folder in the user's home directory. Perhaps somewhere like under /opt instead? XQuartz keeps its stuff there.

Really liking multipass! I've been playing with it a lot lately, but needed to fire up something new in VirtualBox yesterday and thought, "Oh my god, this is taking SOOO long compared to multipass."

[Saviq]

FWIW we're using the QStandardPaths class from Qt, relying on it to tell us the right place to store our data and caches.

It doesn't seem like /opt is returned for any of the options. Have you seen a summary of macOS locations somewhere that might suggest a better place than what we're using now?

Really liking multipass! I've been playing with it a lot lately, but needed to fire up something new in VirtualBox yesterday and thought, "Oh my god, this is taking SOOO long compared to multipass."

That's great! We have some speed-ups up our sleeves, too ;)

[BeauSlim]

OK, having worked on Linux and FreeBSD for years, but really only being an end-user with Apple's FreeBSD/Mach/System7 Frankenstein back-end, I did some research.

The most current, official doc from Apple appears to be here This doc also gives an idea of where Apple thinks unix-style services/daemons should keep their files.

If you want to keep everything hidden where it is and simply add a flag to multipass to disable backups (kind of like what docker for mac does with a check-box in the helper UI), start with https://developer.apple.com/documentation/coreservices/backup_core

Otherwise put the VM images in a directory under the user's home, under ~/Library/, under /Library/, or follow BSD/brew conventions under /usr/local/*. All of these are at least read-accessible so the user can choose to exclude them from Time Machine backups using the standard interface, or to restore them from TM recovery images.

I also did a quick look at where other hypervisors put their stuff. Docker uses Apple's application sandboxing, so I don't think that applies to multipass (?). VirtualBox, vmware Fusion, and Parallels pretty much all put VM images/instances under a directory in the user's home, background daemon binaries and configs under /Library/Application Support/, with custom kernel extensions under /Library/StagedExtensions/.

Speed-ups are always good. I discovered today that Parallels-lite, which used to be free to use for linux VMs, has essentially been discontinued. That pretty much leaves just multipass and VirtualBox as the free options on mac.

[cassiocassio]

+1

~/Library/Application Support [per user stuff] and /Library/Application Support [for all users of an app]

are exactly where mac users expect well behaved applications to keep their 'plumbing' that is (i) not part of the core app bundle and binary that it needs to launch, and (ii) not something that users need to be aware of, feel they have created themselves, files they need to directly interact with [belongs in ~].

It's tricky because ubuntu and BSD-leaning mac users have their own conventions, and brew is a pretty strong influence too.

Using native conventions for Application support means other native software like Migration Assistant will seamlessly move multipass' resources to a brand new mac, as well as Time Machine being able to do the right thing.

[wzhliang]

+1 Even better if it can be configured. MacBooks are not meant to be used for playing with multiple VMs. One has to use some external storage.

[ingramchen]

+1 My time machine goes crazy because I could not exclude multipassd instances...

[ztolley]

I would really appreciate being able to set the location for storage of a vm on Mac so I can store them on attached storage instead of the internal disk that is limited

[michaeldam]

I did the following to quickly move the multipass images to a proper place in order for me to be able to exclude them from Time Machine backups. Same procedure can be used to make multipass run off of an external storage.

First make sure multipass is not running.

On the command line switch to Super User mode. sudo su

Go to directory where the multipass data is located cd /var/root/Library/Application\ Support/

Move the multipass directory to a proper place mv multipassd/ /Library/Application\ Support/

create a symbolic link called multipassd (which we moved in the previous command) and have it point to the new location ln -s /Library/Application\ Support/multipassd/ ./multipassd

We need to do also move the the multipass image cache

Go to directory where the multipass cache is located cd /var/root/Library/Caches

Move the multipass cache directory to a proper place mkdir /Library/Application\ Support/multipassd/Caches mv multipassd/ /Library/Application\ Support/multipassd/Caches

create a symbolic link called multipassd (which we moved in the previous command) and have it point to the new location ln -s /Library/Application\ Support/multipassd/Caches/multipassd/ ./multipassd

done

In time machine I've excluded /Library/Application\ Support/multipassd/vault/instances/ /Library/Application\ Support/multipassd/Caches/multipassd/

Edit: updated the instructions to include the image Cache directory.

Edit2: Solved issue where macOS updates remove the newly introduced multipassd directory in /Library/Caches causing:

1. multipass failing to start and
2. potentially making macOSs crash deamon do overtime, even while sleeping on battery, causing the battery to drain rapidly.

Fix: Use /Library/Application\ Support/multipassd/Caches in favour of /Library/Caches

[goofrider]

A simple workaround is to exclude the folders from Time Machine backup using tmutil. Once you added them to the exclusion list they will show up in the Time Machine prefpane exclusion dialog.

sudo tmutil addexclusion -p /var/root/Library/Application\ Support/multipassd/vault/instances/
sudo tmutil addexclusion -p /var/root/Library/Caches/multipassd/

Don't forget to exclude the cache folder as well.

[EDIT: added -p flag]

[acsr]

A simple workaround is to exclude the folders from Time Machine backup using tmutil. Once you added them to the exclusion list they will show up in the Time Machine prefpane exclusion dialog.

sudo tmutil addexclusion /var/root/Library/Application\ Support/multipassd/vault/instances/
sudo tmutil addexclusion /var/root/Library/Caches/multipassd/

Don't forget to exclude the cache folder as well.

This is not working for me

Commands above execute without warning but nothing shows up in TM SystemPreference Controlpanel and checking with

sudo tmutil isexcluded /var/root/Library/Application\ Support/multipassd/vault/instances/ /var/root/Library/Caches/multipassd/

results in

[Included]    /var/root/Library/Application Support/multipassd/vault/instances
[Included]    /var/root/Library/Caches/multipassd

OS is 10.14.6 Mojave with Multipass 1.3.0

[goofrider]

Can you please try it with the -p flag and see if that works?

sudo tmutil addexclusion -p /var/root/Library/Application\ Support/multipassd/vault/instances/
sudo tmutil addexclusion -p /var/root/Library/Caches/multipassd/

[artificial-aidan]

When I run sudo tmutil isexcluded /var/root/Library/Application\ Support/multipassd/vault/instances/ /var/root/Library/Caches/multipassd/ I get the following:

[Excluded]    /var/root/Library/Application Support/multipassd/vault/instances
[Excluded]    /var/root/Library/Caches/multipassd

But when I do a diff between 2 backups I see the following:

! 48.3G   (size, mtime)         /Volumes/Time Machine Backups/Backups.backupdb/Aidans-MBP/2020-08-31-194554/Macintosh HD - Data/private/var/root/Library/Application Support/multipassd/vault/instances/microk8s-vm/ubuntu-18.04-server-cloudimg-amd64.img

So for some reason it is still being backed up.

[goofrider]

When I run sudo tmutil isexcluded /var/root/Library/Application\ Support/multipassd/vault/instances/ /var/root/Library/Caches/multipassd/ I get the following:

[Excluded]    /var/root/Library/Application Support/multipassd/vault/instances
[Excluded]    /var/root/Library/Caches/multipassd

But when I do a diff between 2 backups I see the following:

! 48.3G   (size, mtime)         /Volumes/Time Machine Backups/Backups.backupdb/Aidans-MBP/2020-08-31-194554/Macintosh HD - Data/private/var/root/Library/Application Support/multipassd/vault/instances/microk8s-vm/ubuntu-18.04-server-cloudimg-amd64.img

So for some reason it is still being backed up.

Did you used the -p flag? I made a mistake and omitted it.

But if tmutil reports they are excluded and they're still backed up then I don't know what to tell u. Might need to seek help in Apple support forum.

I doublechecked my Time Machine backups and indeed vault/ only has multipassd-instance-image-records.json in it. So the they're excluded in TM backup as expected. I'm using Catalina.

[jitinl]

Besides Time machine, it might be a good idea to also exclude the directory from Spotlight indexing, as that at least for me was consuming a lot of CPU when for example installing a package from apt.

[dingsters]

Commenting on old issues, hoping it will help new google searches. I am on Catalina, and above solutions will help. I am also on tight storage, so an external drive is my only way to have multiple Ubuntu using multipass.

The soft link will work, please note, I have initialized 1 instance already when I started looking for the VM files, and google to this page. I move the folder /var/root/Library/Application Support/multipassd to external drive

Before doing this, make sure your instance is "stop" and quit "Multipass" UI located on the task bar icon.

sudo mv /var/root/Library/Application\ Support/multipassd /Volume/EXTERNAL-DRIVE-NAME/Multipass-file/multipassd

Then I made a soft link

sudo ln -s /Volume/EXTERNAL-DRIVE-NAME/Multipass-file/multipassd /var/root/Library/Application\ Support/multipassd

Dont forget to add multipassd from "System Preferences -> Security & Privacy -> Full Disk Access" so that it can access your external drive(s).

Thats it!... You can add the LINK to ignore in Time Machine, I do add it up, even though my whole external drive was already excluded.

For some reason, you need or rather want to use VirtualBox or VM and want to do the same storing that large VM.disk to an external drive, the solution is almost the same.

[mboutet]

I am on Big Sur and performed the steps from https://github.com/canonical/multipass/issues/566#issuecomment-660611725. I see these two folders excluded in Time Machine preferences. However, it seems that Time Machine is still taking backup of the instances anyway. My TM hard drive is filling up pretty quickly. In 3 weeks, I've lost almost 2TiB of space 😞.

Any idea someone? @artificial-aidan, did you manage to definitely exclude the instances from the backups?

[schveiguy]

I'm surprised at the secrecy and obscurity of where you are putting these images. I too am concerned about time machine backups, as I have seen my time machine go from a 4 year history to a 2 week history when I forgot to exclude VM images from backups using vmware fusion. But VMWare has no problem both telling me where the images are, and also putting them in reasonable locations.

I really think you should follow this recommendation, and put these in normal locations for Mac users.

[schveiguy]

BTW, @mboutet I just followed the symlink recommendations, and it seems to be working. I find that much easier to understand. And it doesn't take long to do, just make sure you also do the Caches/multipassd directory also.

[michaeldam]

Glad it worked for you too @schveiguy. I've updated my step-by-step to include the image cache folder. Your update reminded me of that. Thanks!

[sanderdekoning]

Thanks for this @michaeldam ! I couldn't find the paths quickly elsewhere and the ln copy pastes were very handy

[sanderdekoning]

FYI @michaeldam & all here - creating the symlinks as described here https://github.com/canonical/multipass/issues/566#issuecomment-641581396 worked initially but then my multipass became non-functional: every command would suddenly result in Please ensure multipassd is running and /var/run/multipass_socket is accessible - and there did not exist any file at /var/run/multipass_socket. Reinstalling nor rebooting solved my issue. Only when I deleted the symlinks and therefor reverted those changes in https://github.com/canonical/multipass/issues/566#issuecomment-641581396, and then reinstalling multipass, did it work again. Although I cannot pinpoint the cause exactly at the moment of writing I'm guessing it is related to my either installing openvpn with brew, or updating to macOS 12.1.

[michaeldam]

@sanderdekoning Sorry to hear this. Here's my take based on your observations above. Long version first, then the short version: If the period between 'worked initially' and 'became non-functional' is an extended period possibly including reboots events, I lean towards the suggestion that the #566 step led to a stable situation. This would imply that 'something happened', leading to the erroneous state you describe. I'm glad to hear you were able to recover from this state by undoing the mv's and ln's and reinstalling multipass. I lean towards thinking the reinstall fixed the issue. The undoing of the mv's and ln's -which are quite possibly a prereq to a successful reinstallation of multipass- are less likely to have played a role in fixing your issue, I think, based on what I understand from the situation you describe.

Installing OpenVPN is non-invasive to multipass, as far as I know. I'd be very disappointed if moving to macOS 12.1 made this issue occur. In either case, the #566 would have worked before and became non-functional after that. Not knowing for sure that multipass started failing right after either of these installations, suggest that multipass has only been used intermittently over an extended period. That would suggest many things could have happened over an extended period of time that led to the issue you describe.

Short version: Symbolic links are transparently implemented in the filesystem, and I can't think of any scenario involving these that lead to the issue you describe. Glad you recovered though! You can redo #566 now. ;-)

PS It can be frustrating not knowing the root cause after fixing an issue by simply starting over. But in this instance, and provided it isn't a pattern, I'd move on.

[sanderdekoning]

@michaeldam thank you, I really appreciate your time and thoroughness. Indeed I could not pinpoint it exactly sadly, however I'm fairly confident either of the two described actions cause "some" issue. More people seem to have related issues to this change, such as https://github.com/canonical/multipass/issues/1437 (https://github.com/canonical/multipass/issues/1437#issuecomment-619539276). Additionally, according to Canonical, VPNs are known to possibly interfere somehow: https://multipass.run/docs/troubleshooting-networking-on-macos - hence I thought it was worth mentioning it's possible the issue I had coincided with installing/using openvpn (client). There is a (more plausible?) relation with macOS 12.1. The fact remains that reinstalling multipass did not solve any issue, neither did rebooting or any combination of those two; however removing the symlinks and then installing multipass did resolve it. Of course I don't want to spread misinformation, I thought these findings might help someone who comes across this thread.

[michaeldam]

...The fact remains that reinstalling multipass did not solve any issue, neither did rebooting or any combination of those two; however removing the symlinks and then installing multipass did resolve it. @sanderdekoning My bad, I had not read from your report you'd also reinstalled multipass with the symlinks in place. I can indeed see (thx for this link) that VPN network routing can cause issues with VMs or containers themselves. Given the case as you describe it I have a hard time seeing the connection but yeah perhaps multipass itself could get confused. Should be reproducible. ...Of course I don't want to spread misinformation, I thought these findings might help someone who comes across this thread. Of course you don't. Yours was a legitimate call out. All of us here can only spot patterns provided we share as you did. Thanks for that! Odd case though. I'm still wondering.

[sanderdekoning]

@ricab is there anything you could say regarding this issue and any possible relation to your comment https://github.com/canonical/multipass/issues/1437#issuecomment-967017377 and https://github.com/canonical/multipass/issues/566#issuecomment-992969309?

@michaeldam Indeed! I'll make sure to keep you and the others up to date on any new findings

[schveiguy]

I can confirm that I updated to 12.1, and my multipass is no longer working.

I see "Failure Retrieving Instances" in the menu. I'll try reversing the symlink steps and see if that helps.

[schveiguy]

Some more info:

/Library/Caches/multipassd disappeared. When I removed the symlink in /var/root/Library/Caches, then multipass created the directory automatically. For sure there is something funky going on.

After moving the cache directory over, and symlinking again (and rebooting), it is fixed, I can now start and connect to my instance.

This might not be a permanent solution. Perhaps putting the directory to symlink to in a different location might be in order, it appears that the OS might remove the /Library/Caches directories.

[michaeldam]

@schveiguy I was able to reproduce this on another machine. Init: macOS < 12.1; multipass installed; multipass directories moved according to #566 step: upgrade to macOS 12.1 result: multipass misbehaving fix: cd /Library/Caches/ mkdir multipassd logoff/logon or restart if you feel like it for me multipass works again now apparently (some?) /Library/Caches/ are removed during upgrade. Thinking of it it might make sense. However then another target dir for the multipass caches is preferable. Suggestions anyone? What would be a 'right' and 'proper' place, untouched by macOS updates?

Edit: @sanderdekoning 👆 Edit2: typo

[schveiguy]

mkdir multipass

should be multipassd

[schveiguy]

logoff/logon or restart if you feel like it

Just another piece of data, restarting multipassd did not work for me, it wouldn't start my instance. I had to reboot (Well, maybe just logoff would have worked, I'm not sure).

A question about that, is selecting "quit" from the multipass menu enough to get it to stop?

[michaeldam]

@michaeldam Indeed! I'll make sure to keep you and the others up to date on any new findings @sanderdekoning Well done. Turns out you were #1 in reporting this particular issue and it became a pattern. And I learned something :-)

[michaeldam]

A question about that, is selecting "quit" from the multipass menu enough to get it to stop?

@schveiguy My experience is that when multipass is running normally, 'quit' will do the trick. However, after the move to 12.1, I was not able to close multipass this way (it became messy with skippy balls and respawning itself when killing multipass using the Activity Monitor). Eventually I was able to logoff and that was sufficient. My findings my very well have been a local experience only. But I would think that after the recreate of the multipassd directory, a restart of multipassd one way or the other, is required. Rebooting macOS will definitely give one a cleaner start over logoff/logon.

[sanderdekoning]

Glad we found the cause/that I was able to contribute! @michaeldam

[ricab]

@ricab is there anything you could say regarding this issue and any possible relation to your comment #1437 (comment) and #566 (comment)?

Hi @sanderdekoning, sorry for the delay. Answer here.

[schveiguy]

Just want to clarify, the fix for #1437 doesn't change the situation for this bug, right? Or is there a path to easily excluding VM images using that technique?

[Saviq]

@schveiguy see #1789 - you can change where Multipass stores its data and have that excluded by TimeMachine, if that's what you're asking for.

[schveiguy]

Well yeah the whole point of this thread is to request putting the image in a more standard directory! I’m really not sure how that xml plist thing works but I’ll give it a try later. If that works we have no need for symlinks

[michaeldam]

...What would be a 'right' and 'proper' place, untouched by macOS updates?

I've updated #566 to move away from /Library/Cache which is cleared during macOS updates causing multipass to fail. Bottom line: the creation of directory /Library/Application\ Support/multipassd/Caches and using it in favour of /Library/Caches.

[earendildev]

566 worked for me - MacOS Monterey

If you installed multipass through Brew, you installed the CLI version.

Make sure to first stop all instances, then sudo launchctl unload /Library/LaunchDaemons/com.canonical.multipassd.plist and after completing the steps sudo launchctl load /Library/LaunchDaemons/com.canonical.multipassd.plist

Wait for about 20sec to a min, then multipass info --all to check and view all your instances.

[joes]

I created /Library/LaunchDaemons/setenv.MULTIPASS_STORAGE.plist and set the storage path to an external drive:

/bin/launchctl setenv MULTIPASS_STORAGE "/Volumes/Transcend 960/multipass/storage" 

I also ensured that the directory was owned and readable/writeable by root:wheel.

However, multipassd fails to restart. In /Library/Logs/Multipass/multipassd.log I can see this error repeating:

[2022-08-21T08:35:36.135] [error] [daemon] Caught an unhandled exception: unable to create directory '/Volumes/Transcend 960/multipass/storage/cache'

What am I missing? Or is setting the MULTIPASS_STORAGE to an external drive not supported due to some reason?

[joes]

What am I missing? Or is setting the MULTIPASS_STORAGE to an external drive not supported due to some reason?

Solution:

I needed to allow multipassd "System Preferences -> Security & Privacy -> Full Disk Access" so that it can access external drive(s).

Now I only need to figure out how to make multipassd/launchctl wait for the external drive to be available when restarting the computer.

[kameit00]

:-( Seems like Vagrant is a nicer solution for VMs, if I want to back up my VMs nicely without much hassle. Or is there a configuration option now? Not using "Application Support" feels just wrong...

[sietsevanderbom]

Now I only need to figure out how to make multipassd/launchctl wait for the external drive to be available when restarting the computer.

@joes Figured that out?

[joes]

Now I only need to figure out how to make multipassd/launchctl wait for the external drive to be available when restarting the computer.

@joes Figured that out?

@sietsevanderbom

No, I did not. Multipassd would always start writing to an ”unmounted” path in ’/Volumes’ and the hard drive would get mounted on another ”free” subdirectory in ’/Volumes’.

[phil-m]

A simple workaround is to exclude the folders from Time Machine backup using tmutil. Once you added them to the exclusion list they will show up in the Time Machine prefpane exclusion dialog.

sudo tmutil addexclusion -p /var/root/Library/Application\ Support/multipassd/vault/instances/
sudo tmutil addexclusion -p /var/root/Library/Caches/multipassd/

Don't forget to exclude the cache folder as well.

[EDIT: added -p flag]

For anyone looking to just exclude the default paths for macOS Ventura 13.4 when using the default qemu driver, I had to tweak the vault path to be qemu specific. If you're running VirtualBox the path will likely change again (untested).

I also had to prefix both paths with /System/Volumes/Data/private/var/ otherwise the tmutil isexcluded results would always show [Included] along with that extended full path in its results.

Also had to grant Terminal (or iTerm in my case) 'Full Disk Access' to be able to run the commands.

Final commands I ended up running (thanks @goofrider for the original suggestion):

sudo tmutil addexclusion -p /System/Volumes/Data/private/var/root/Library/Application\ Support/multipassd/qemu/vault/instances/
sudo tmutil addexclusion -p /System/Volumes/Data/private/var/root/Library/Caches/multipassd/

Then to check (seem to be able to use the original or newer extended path when querying it):

sudo tmutil isexcluded /var/root/Library/Application\ Support/multipassd/qemu/vault/instances/ /var/root/Library/Caches/multipassd/

Should show [Excluded] next to both if it worked, e.g:

[Excluded]    /System/Volumes/Data/private/var/root/Library/Application Support/multipassd/qemu/vault/instances
[Excluded]    /System/Volumes/Data/private/var/root/Library/Caches/multipassd

I will look to move the path as more recent comments above suggest when I have more time, just needed a quick fix to stop Time Machine eating up space & taking ages over Wifi for now.

As the original issue & comments suggested, it would be really nice to use easily accessible dirs by default on macOS, or make it easy & obvious to change it via the GUI, or at least perhaps just warn us on first run/use perhaps, so we don't create a load of VMs & fill up Time Machine & only notice later as I did. Especially as Time Machine no longer supports deleting/purging files once backed up!

Saying all that, Multipass has been an amazingly helpful & well crafted tool, & appreciate macOS is only one target OS, big thanks to all involved!!

[EricPengShuai]

I'm new to multipass on Mac M chip. I create an instance called ubuntu2204 by multipass launch ubuntu2204. I wonder where is the instance file.

Why didn't I find Library/Application Support/Multipassd folder. There is only Library/Application Support/Multipass dir. (I install multipass by multipass-1.12.2+mac-Darwin.pkg in the release page.