search

utmapp / UTM

Virtual machines for iOS and macOS
https://getutm.app
Apache License 2.0
26.95k stars 1.35k forks source link

Run .utm from anywhere #1205

Closed phsimons closed 2 years ago

phsimons commented 3 years ago

It would be nice to be able to store the .utm VMs some where else than ~\Documents, for exemple an attached disk.

Version 2.0.13(13)

thebitstick commented 3 years ago

This is especially important when using iCloud and it tries to backup .utm VMs :/

stefandesu commented 3 years ago

Yeah, I just had the same thought when iCloud started uploading the VM...

thebitstick commented 3 years ago

Workaround: quit UTM, move your .utm bundles elsewhere, and the symlink them to your Documents folder. This will need to be done manually for every VM you create, but at least you don't use up your iCloud storage.

ty-yqs commented 3 years ago

Yeah, I just had the same thought when iCloud started uploading the VM...

But .utm files are usually very large. So if you don’t download .utm file in iCloud, it will become very slow to open this file.

stefandesu commented 3 years ago

Yeah, I just had the same thought when iCloud started uploading the VM...

But .utm files are usually very large. So if you don’t download .utm file in iCloud, it will become very slow to open this file.

The issue is about not having the files in iCloud in the first place. ;) Simply said, we want to be able to choose where the VMs are stored because the Documents folder is often synced via iCloud Drive.

sulliops commented 3 years ago

Definitely want this feature, running VMs off external drives is the biggest reason why for me.

Workaround: quit UTM, move your .utm bundles elsewhere, and the symlink them to your Documents folder. This will need to be done manually for every VM you create, but at least you don't use up your iCloud storage.

Just for anyone that might be confused as to how to do this, the command for Terminal is as follows:

ln -s /path/to/original /path/to/link
jpc commented 3 years ago

Hi,

In the newest (non App Store) version the symlink works only if the VM is located on the system disk. If I move the .utm bundle to an external USB disk (APFS formatted) and create the symlink, the VM does not show in the UI at all.

I also tried to symlink the qcow2 files inside the .utm bundle but I get errors when trying to start the VM:

libqemu-aarch64-softmmu.utm.dylib: -drive if=virtio,media=disk,id=drive0,file=/Users/jpc/Library/Containers/com.utmapp.UTM/Data/Documents/Debian ARM.utm/Images/debian.qcow2,cache=writethrough: Could not open '/Users/jpc/Library/Containers/com.utmapp.UTM/Data/Documents/Debian ARM.utm/Images/debian.qcow2': Operation not permitted

I've manually granted the UTM app "Full disk access" but it did not help.

osy commented 3 years ago

That workaround is outdated from back when UTM was not a sandboxed app.

sulliops commented 3 years ago

If I move the .utm bundle to an external USB disk (APFS formatted) and create the symlink, the VM does not show in the UI at all.

I was (am) having the same problem with older versions, it's definitely not new. Would be really nice if we could just add an entry to the UTM GUI and tell it where to find the .utm.

osy commented 3 years ago

Yeah that’s why this issue exists as a feature proposal...

artyomxx commented 3 years ago

It's a shame that nowadays this kind of functionality is not being built-in into trendy VM software from the very start. It's essential.

osy commented 3 years ago

It's a shame that nowadays this kind of functionality is not being built-in into trendy VM software from the very start. It's essential.

It’s a shame Apple’s apis don’t allow for this to be done easily. It’s open source though why don’t you give it a shot and send a PR once it’s done. We’d all be very thankful 😉

thebitstick commented 3 years ago

It's a shame that nowadays this kind of functionality is not being built-in into trendy VM software from the very start. It's essential.

Since when did UTM have a price tag. Such entitlement 😔

fulldecent commented 3 years ago

Here are some behavior changes that could be in-scope for this issue:

Discussion:

Personally I think of .utm files as files that I control. I can torrent them, share them, back them up, delete them, copy them somewhere. And since they are big, I can run them in-place from an external drive. Also, I assume VMs specifically cause a lot or wear on a hard disk so I assume keeping them off a soldered-on main drive is a good idea.

Currently, UTM stores all files in a hidden folder that can be many many gigabytes. This is too easy to forget about. I feel that only preferences and minor configuration items should be stored in hidden folders.

prr643 commented 3 years ago

I'm sorry, but I see the inability to store utm files easily where ever you like, especially to external devices, as a huge flaw in the UTM software. This is especially true for software designed for modern Macs, where small internal SSD sizes are unfortunately common on most non-custom ordered machines. The ability to easily run a VM from an external device is not only needed, but in my opinion essential for virtualization software for a Mac.

kevingoh commented 3 years ago

I have just experimented the solution as suggested by this post and it worked (i.e. I could run .utm from my external drive)

harperjohn commented 3 years ago

Thanks @kevingoh !

I (eventually) got this to work too!!

My initial struggle was that I tried to mount the sparse image to the "UTM" (which is actually "com.utmapp.UTM") and then UTM just seemed to ignore the link and create a new "UTM" folder.

I then mounted a sparseimage to the "Data" sub-folder, and this seemed to work fine!

Did you got it to work at the UTM folder level ? (I do think it is cleaner/neater to link at the UTM level... but I am also quite fine with my link at the "Data" folder level, as it seems to be working fine)

xros commented 3 years ago

I have just experimented the solution as suggested by this post and it worked (i.e. I could run .utm from my external drive)

Thanks. This is current method to run .utm vms from external drive. It works. I create a shell script to run it automatically.

echo "[+] ----- Mounting to /Users/alex/Library/Containers/com.utmapp.UTM/Data/Documents ----"
hdiutil attach /Volumes/sn550-1t/vms_parallels_arm64/utm_vms/sparse_image_as_virtual_disk_for_utm.sparseimage -mountpoint /Users/alex/Library/Containers/com.utmapp.UTM/Data/Documents
echo "[+] ----- Finished mounting ---"

It's not perfect but it works.

mertsaner commented 3 years ago

Hi, I try to run UTM via an external hard drive, when I try allies on the hard drive and try to import it to UTM, it copies the original .utm file to document, however, I would like it to run on an External drive, not in the Macbook. How can I do that? Thanks for any help,

mertsaner commented 3 years ago

Workaround: quit UTM, move your .utm bundles elsewhere, and the symlink them to your Documents folder. This will need to be done manually for every VM you create, but at least you don't use up your iCloud storage.

It did not work with me, can you describe how you do that? It does not appear my UTM when I make a symlink of the UTM file that is on the external disk.

mertsaner commented 3 years ago

I have just experimented the solution as suggested by this post and it worked (i.e. I could run .utm from my external drive)

Thanks. This is current method to run .utm vms from external drive. It works. I create a shell script to run it automatically.

echo "[+] ----- Mounting to /Users/alex/Library/Containers/com.utmapp.UTM/Data/Documents ----"
hdiutil attach /Volumes/sn550-1t/vms_parallels_arm64/utm_vms/sparse_image_as_virtual_disk_for_utm.sparseimage -mountpoint /Users/alex/Library/Containers/com.utmapp.UTM/Data/Documents
echo "[+] ----- Finished mounting ---"

It's not perfect but it works.

When I try this I got the error “com.utmapp.UTM” couldn’t be", any suggestions?

Dominik-1980 commented 2 years ago

Would be great if there is any solution, because my Mac Mini M1 only has 256GB internal SSD...

mertsaner commented 2 years ago

The link above worked for me afterwards (after spending time and figuring out what is going on exactly) , so it is doable for sure, but M1 does not support Ubuntu 18.04 natively so it is not practical for some use cases but 20.04 works like a charm.

xros commented 2 years ago

I have just experimented the solution as suggested by this post and it worked (i.e. I could run .utm from my external drive)

Thanks. This is current method to run .utm vms from external drive. It works. I create a shell script to run it automatically.

echo "[+] ----- Mounting to /Users/alex/Library/Containers/com.utmapp.UTM/Data/Documents ----"
hdiutil attach /Volumes/sn550-1t/vms_parallels_arm64/utm_vms/sparse_image_as_virtual_disk_for_utm.sparseimage -mountpoint /Users/alex/Library/Containers/com.utmapp.UTM/Data/Documents
echo "[+] ----- Finished mounting ---"

It's not perfect but it works.

When I try this I got the error “com.utmapp.UTM” couldn’t be", any suggestions?

Firstly you need to make a file such as sparse_image_as_virtual_disk_for_utm.sparseimage from this post Later, you need to modify the codes I wrote. Change it to your own username on MacOS.

/Users/alex/Library/Containers/com.utmapp.UTM/Data/Documents to /Users/your_user_name/Library/Containers/com.utmapp.UTM/Data/Documents

Don't forget to change your folder's path, where you want to be mounted to on your external SSD. E.g. I was using a SN550 1T hard drive.

yasinusta commented 2 years ago

Can someone make a video about it so we can run it on our external drive? I am getting a errors and I probably doing something wrong. No idea how to run shell script on my mac.

Madhawa97 commented 2 years ago

@xros So the hdiutil attach command needs to be run, each time I wanna attach the sparse image. right? Assuming Yes, I created an alias in .zshrc works as expected. Thank you.

xros commented 2 years ago

@xros So the hdiutil attach command needs to be run, each time I wanna attach the sparse image. right? Assuming Yes, I created a alias in .zshrc works as expected. Thank you.

Yes currently you have to run it everytime when you want to use UTM.

I glad that helps you. 😉

adespoton commented 2 years ago

For those of you writing external scripts to mount images for use in UTM... ...why not just brew install qemu and copy the command-line flags from UTM to run the same configuration from a .sh file? Then you get full access to the monitor as well, and can access stuff that is unavailable inside a sandbox. By default it doesn't come with SPICE, so you'd also have to adjust your brew config to set the SPICE flags if you want that integration.

To me the point of UTM is that it's something contained that will be allowed on the app store and provides simple, visual "yes this just works" options for easy use. So any solutions provided should be things that work with sandboxed apps.

Maybe it's just a case of entitlements being set correctly, so there's access to the ~/Documents folder?

ipriyam26 commented 2 years ago

@adespoton can you please give some steps on how to do it exactly

adespoton commented 2 years ago

@ipriyam26 do which exactly? Run QEMU without UTM? Or how to add entitlements to UTM so that it can use the ~/Documents/ folder instead of the ~/Library/Containers/ folder?

For the first, take the UTM QEMU export string and strip out all the SPICE components, install homebrew on your Mac, and run brew install qemu. Then from the terminal (or a .sh or .command file) run the string UTM generated, but point the drive image at whatever location you want instead of the copy in the UTM container folder.

If you want SPICE integration, you'll have to go to the QEMU website, download QEMU, go to the SPICE website, download the SPICE source, and compile QEMU with the SPICE flags set. If there's already a keg set up on homebrew for SPICE-enabled QEMU, I don't know about it.

osy commented 2 years ago

Implemented as part of #132

gitKearney commented 1 year ago

I figured I'd post what I consider potato head instructions, cause creating a sparse image wasn't easy

STEP 1 - create the sparse image

Open Disk Utility. Create a new Blank Image

Save As: Documents Name: Documents Size 32 GB Image Format: Sparse Disk Image

STEP 2

QUIT UTM

STEP 3 -

EJECT THE SPARSE IMAGE FROM DISK UTILITY

STEP 3

MOUNT THAT SPARSE IMAGE

STEP 4 - PROFIT!

now you should be able to start UTM and your VMs should show up. Be mindful, they are a bit slower but not by much

xros commented 1 year ago

I figured I'd post what I consider potato head instructions, cause creating a sparse image wasn't easy

STEP 1 - create the sparse image

Open Disk Utility. Create a new Blank Image

Save As: Documents Name: Documents Size 32 GB Image Format: Sparse Disk Image

STEP 2

QUIT UTM

STEP 3 -

EJECT THE SPARSE IMAGE FROM DISK UTILITY

STEP 3

MOUNT THAT SPARSE IMAGE

  • rename the existing UTM/Data/Documents directory
  • hdiutil attach /Volumes/MY_EXTERNAL_SSD/Documents.sparseimage -mountpoint ~/Library/Containers/com.utmapp.UTM/Data/Documents
  • move the existing Public and .utm files to the directory ~/Library/Containers/com.utmapp.UTM/Data/Documents

STEP 4 - PROFIT!

now you should be able to start UTM and your VMs should show up. Be mindful, they are a bit slower but not by much

Exactly! Sorry I didn't write it clearly in my last post.

Hope UTM can really consider to customize the file path of your VMs once AGAIN!

kingson87 commented 1 year ago

hdiutil attach /Volumes/SSD/UTM.sparseimage -mountpoint ~/Library/Containers/com.utmapp.UTM/

This command worked for me on M1 mac and copying the data folder as a whole

anuprshetty commented 1 year ago

Hey guys, I got a simple straight forward solution to this.

I am giving the steps from the beginning:

install UTM on Mac create required VMs in the UTM following the standard procedures. --> at this point, created VMs complete data is stored at location --> ~/Library/Containers/com.utmapp.UTM/Data/Documents/ --> for example, in my case, i have created kali and parrot VMs. So data location of these VMs will be, --> i) ~/Library/Containers/com.utmapp.UTM/Data/Documents/kali.utm (contains OS + data) --> ii) ~/Library/Containers/com.utmapp.UTM/Data/Documents/parrot.utm (contains OS + data) --> So ideally you just need to move above 2 utm files to your desired destination location. But don't do it manually. Let UTM software do the job. See next step. Insert your external harddrive to Mac. Now open your UTM dashboard(App). In the left side you see all the VMs. Right click on each VM, then select "move" option. After that select the destination location as your external harddrive location. Then click "confirm". Now you have moved your VM to your destination location. Also VM data at location ~/Library/Containers/com.utmapp.UTM/Data/Documents/ is removed. Repeat the same for all the VMs. NOTE: UTM software simply needs an .utm file. So you can manually keep that file anywhere you want. Open UTM software dashboard. Then open finder and double click on the file which you have copied to open in UTM software. That's it.

Cons: You have to do this manually for each VM you create. Anyways we achieved what we want. Peace :)

gitKearney commented 1 year ago

@anuprshetty this must be new functionality that osy got working. thanks for posting. I'll try this out

malhal commented 1 year ago

I tried the right-click on the VM and move to external SSD but the app just hangs and uses 100% CPU so I killed it. They need to get the copy in a background thread and show progress in the UI.

cw1nte commented 3 months ago

Move (=copy) sparse images can get super-slow, like really days I hear. Has anyone suffered that? I've had it slow but not THAT slow, mostly I have had success without using the copy routines coded within UTM [ I was OK moving uVM folders using Finder]. I believe someone (osy?) had maybe forked in UTM code to fix the slow copying.

Also finding it very sketchy looking at file sizes and their exact nature: is it macOS sparse, clone (?which may induce CoW behaviour on top of qcow2 format which again uses CoW). Often the package (eg reveal package contents) in macOS may be much larger or smaller than the apparent size of the files within it. Not enough time to really study this issue, yet...

I also suspect Finder moves between drives may end up with something UTM will find/treat/result as a Clone (in macSO view) - which AFAIK should not be the case (at least on initial copy). YMMV I'm interested to hear...

malhal is right about the copying issue, I run iostat in terminal to watch disc activity when UTM does any disc thing - this allows me to see how it progresses, size and number of transfers, when it ends, how long it took...