search

utmapp / UTM

Virtual machines for iOS and macOS
https://getutm.app
Apache License 2.0
26.6k stars 1.33k forks source link

"macOS style" bootable ISOs can't boot in UTM virtual machines #3904

Closed steven-michaud closed 2 years ago

steven-michaud commented 2 years ago

I can't get "macOS style" bootable ISOs to boot in any kind of UTM virtual machine -- "macOS 12+" or "Other".

In a macOS 12.3.1 VM (created and run on an Apple Silicon Mac Mini running macOS 12.3.1) I add a bootable ISO as an external drive, then "move it up" (I assume this is to put the drive earlier in the boot order). Then when I start the VM it chews CPU indefinitely, without ever actually booting.

On an Intel machine (a MacBook Pro) running macOS 12.3.1 I attempt to create a VM of type "Other" and set the "Boot ISO Image" to a bootable ISO. When I start the VM it fails to boot from the ISO -- it falls through to the "UEFI Interactive Shell".

This is using UTM 3.1.5, downloaded from https://mac.getutm.app/.

Is this a bug? By design? Apple's fault? Is there anything UTM can do about it?

For reference, this is how I create my "macOS style" bootable ISO. I can use VMware Fusion (on an Intel Mac) to create a VM from it, so I know it really is bootable.

  1. Download the full installer:

    softwareupdate --fetch-full-installer --full-installer-version 12.3.1

  2. Create a sparse image to work with:

    hdiutil create -o blah -size 15G -volname blah -layout SPUD -type SPARSE -fs HFS+J

  3. Mount this sparse image:

    hdiutil attach blah.sparseimage -noverify -mountpoint /Volumes/blah

  4. Use the full installer's createinstallmedia to write a bootable installer to the sparse image:

    sudo /Applications/Install\ macOS\ Monterey.app/Contents/Resources/createinstallmedia --volume /Volumes/blah

  5. Detach the bootable installer:

    hdiutil detach -force /Volumes/Install\ macOS\ Monterey

  6. Create the bootable ISO:

    hdiutil convert blah.sparseimage -format UDTO -o macOS_Monterey_12.3.1_Installer

  7. Rename the ISO:

    mv macOS_Monterey_12.3.1_Installer.cdr macOS_Monterey_12.3.1_Installer.iso

watersb commented 2 years ago

I can't get "macOS style" bootable ISOs to boot in any kind of UTM virtual machine -- "macOS 12+" or "Other".

@steven-michaud - Thanks for your very interesting HookCase tool!

The articles on Eclectic Light regarding Apple Silicon Boot might be of interest here.

Back to UTM, there is some interesting weirdness here when I tried to add a disk image as an External Drive. I kept the image format as SPARSE, and it appeared an an internal drive to the macOS guest VM, but it was not recognizable in that guest.

Converting it to another format, such as UDRW or UDTO, the disk image did not appear at all.

Just for fun, I wondered what qemu-img would think about these disk images. The disk image used by UTM itself for the macOS VM is not exactly a standard diskarbitrationd style image, although it does mount via hdiutil:

% hdiutil attach /Users/bwaters/Library/Containers/com.utmapp.UTM/Data/Documents/macOS.utm/Data/disk0.img 
There may be a problem with this disk image. Are you sure you want to open it?
[Opening this disk image may make your computer less secure or cause other problems.]
Attach? (Y/N) y

You get the standard Apple Silicon disk volume layout, but I didn't poke around on this image, given the warnings...

% hdiutil info
framework       : 623.100.1
driver          : 623.100.1
images          : 1
================================================
image-path      : /Users/me/Library/Containers/com.utmapp.UTM/Data/Documents/macOS.utm/Data/disk0.img
image-alias     : /Users/me/Library/Containers/com.utmapp.UTM/Data/Documents/macOS.utm/Data/disk0.img
shadow-path     : <none>
icon-path       : /System/Library/PrivateFrameworks/DiskImages.framework/Resources/CDiskImage.icns
image-type      : read/write
system-image    : false
blockcount      : 134217728
blocksize       : 512
writeable       : TRUE
autodiskmount   : TRUE
removable       : TRUE
image-encrypted : false
mounting user   : me
mounting mode   : <unknown>
process ID      : 45057
framework name  : DiskImages
/dev/disk4  GUID_partition_scheme   
/dev/disk4s1    69646961-6700-11AA-AA11-00306543ECAC    
/dev/disk6  EF57347C-0000-11AA-AA11-00306543ECAC    
/dev/disk6s1    41504653-0000-11AA-AA11-00306543ECAC    /Volumes/iSCPreboot
/dev/disk6s2    41504653-0000-11AA-AA11-00306543ECAC    /Volumes/xART
/dev/disk6s3    41504653-0000-11AA-AA11-00306543ECAC    /Volumes/Hardware
/dev/disk6s4    41504653-0000-11AA-AA11-00306543ECAC    /Volumes/Recovery
/dev/disk4s2    7C3457EF-0000-11AA-AA11-00306543ECAC    
/dev/disk8  EF57347C-0000-11AA-AA11-00306543ECAC    
/dev/disk8s1    41504653-0000-11AA-AA11-00306543ECAC    /Volumes/Macintosh HD
/dev/disk8s2    41504653-0000-11AA-AA11-00306543ECAC    /Volumes/Preboot
/dev/disk8s3    41504653-0000-11AA-AA11-00306543ECAC    /Volumes/Recovery 2
/dev/disk8s4    41504653-0000-11AA-AA11-00306543ECAC    /Volumes/Update 1
/dev/disk8s5    41504653-0000-11AA-AA11-00306543ECAC    
/dev/disk8s6    41504653-0000-11AA-AA11-00306543ECAC    /Volumes/VM
/dev/disk4s3    52637672-7900-11AA-AA11-00306543ECAC    
/dev/disk7  EF57347C-0000-11AA-AA11-00306543ECAC    
/dev/disk7s1    41504653-0000-11AA-AA11-00306543ECAC    /Volumes/Recovery 1
/dev/disk7s2    41504653-0000-11AA-AA11-00306543ECAC    /Volumes/Update

Finally, I wanted to see what UTM's qemu-image utility would think about this image. I realize that the macOS guest virtualization is handled by virtualizationFramework, and isn't emulated by QEMU...

But I hit another, unexpected bump:

% /Applications/UTM.app/Contents/Frameworks/qemu-img.framework/Versions/A/qemu-img
zsh: exec format error: /Applications/UTM.app/Contents/Frameworks/qemu-img.framework/Versions/A/qemu-img

Yow. "exec format error"?

% file /Applications/UTM.app/Contents/Frameworks/qemu-img.framework/Versions/A/qemu-img 
/Applications/UTM.app/Contents/Frameworks/qemu-img.framework/Versions/A/qemu-img: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit dynamically linked shared library x86_64
- Mach-O 64-bit dynamically linked shared library x86_64] [arm64]
/Applications/UTM.app/Contents/Frameworks/qemu-img.framework/Versions/A/qemu-img (for architecture x86_64): Mach-O 64-bit dynamically linked shared library x86_64
/Applications/UTM.app/Contents/Frameworks/qemu-img.framework/Versions/A/qemu-img (for architecture arm64):  Mach-O 64-bit dynamically linked shared library arm64

Ok, sorry, not helpful.

But in summary, getting disk images attached to a UTM macOS guest VM is not (yet?) working as I might expect from the other path via QEMU.

steven-michaud commented 2 years ago

I have no trouble adding my bootable ISO as an "external drive" to my M1 macOS 12.3.1 guest running on an M1 macOS 12.3.1 host. It shows up (like you said) as an internal drive. But I'm able to navigate it, and even to start the installer on it (though of course I quit before anything's actually installed). "New External Drive", for UTM, seems to mean "attach a drive image from somewhere outside the default location of ~/Library/Containers/com.utmapp.UTM". My problem is that I can't boot from it.

The tests you performed on your sparse image may have damaged it before you converted it to UDTO format. That may explain the troubles you had with your ISO.

Thanks for your link to those Eclectic Light articles. I may find in them some way to trick the boot loader into failing over to the recovery partition. Assuming, of course, that the guest actually has one ...

The whole reason I want to boot from an Apple bootable installer is to load its recovery partition, so that I can turn off SIP in the guest, so that I can use it to work on porting HookCase to Apple Silicon. I'll be happy if I can find another way to do this.

@steven-michaud - Thanks for your very interesting HookCase tool!

You're most welcome :-)

steven-michaud commented 2 years ago

I may find in them some way to trick the boot loader into failing over to the recovery partition.

I did it! It was absurdly simple!

  1. Create a new drive and start the guest VM.

  2. Just after you log in, you'll be warned that "the disk you attached is not readable by this computer". Choose to initialize it, then choose "Erase". As best I can tell, it doesn't matter which format you choose.

  3. Shut down the guest, then "Move Up" the new drive (so that it's the first in the boot order).

  4. Restart the guest VM. It will fail to boot.

  5. Close the guest VM's window, which will kill the VM.

  6. Restore the original boot order (and possibly delete the drive you created in step 1), then restart the guest VM.

  7. As it starts up, you'll be told that "authentication is required to verify startup disk", and prompted for a password.

  8. Choose "Cancel", then choose "Startup Disk".

  9. Quit "Startup Disk". You'll end up in the Recovery utility!

This resolves my problem, but doesn't resolve this bug. So I'll leave it open. But fixing it has now become much less urgent, at least for me.

steven-michaud commented 2 years ago

I've done some more testing, with a LaCie 1TB external SSD that has a Thunderbolt 3 connection.

First I used createinstallmedia to install a macOS 12.3.1 bootable installer on it. I found it would only boot on Intel Macs -- not on Apple Silicon ones. createinstallmedia only works with drives formatted with the HFS+J file system. I suspect this is what makes my external SSD's bootable installer unbootable on Apple Silicon machines.

Then I used macOS 12.3.1 recovery partitions on an Apple Silicon Mac and an Intel Mac to install the macOS 12.3.1 OS on my external SSD. (I did this once on the Apple Silicon Mac, once again on the Intel Mac, and once yet again on the Apple Silicon Mac.) At the end of this, my external SSD was bootable on both my Apple Silicon Mac and my Intel Mac. On installing macOS 10.15, 11 or 12, the target drive/partition gets reformatted to APFS, if need be. This, I suspect, is what made it bootable on both kinds of hardware.

This doesn't entirely explain the problems I reported in my first comment. UTM can't boot from my bootable ISO on either kind of hardware. But it may provide a partial explanation.

watersb commented 2 years ago

Wow, very nice!

Thanks!

On Apr 17, 2022, at 11:59 AM, Steven Michaud @.***> wrote:

 I may find in them some way to trick the boot loader into failing over to the recovery partition.

I did it! It was absurdly simple!

Create a new drive and start the guest VM.

Just after you log in, you'll be warned that "the disk you attached is not readable by this computer". Choose to initialize it, then choose "Erase". As best I can tell, it doesn't matter which format you choose.

Shut down the guest, then "Move Up" the new drive (so that it's the first in the boot order).

Restart the guest VM. It will fail to boot.

Close the guest VM's window, which will kill the VM.

Restore the original boot order (and possibly delete the drive you created in step 1), then restart the guest VM.

As it starts up, you'll be told that "authentication is required to verify startup disk", and prompted for a password.

Choose "Cancel", then choose "Startup Disk".

Quit "Startup Disk". You'll end up in the Recovery utility!

This resolves my problem, but doesn't resolve this bug. So I'll leave it open. But fixing it has now become much less urgent, at least for me.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.

adespoton commented 2 years ago

I may have some information to help explain this (but it won't help fix the "bug" here). Because AARCH64 uses the internal nVME SSD to store the firmware as well as the EFI and regular disk partitions, a properly formatted disk image needs to be available to check the firmware partition, and the virtual machine needs to think it's the internal one. This same issue exists on bare metal hardware: if you don't have an internal firmware partition available, an external ISO image won't boot on an M1 device.

What you've done here is successfully partition an image so that it's got the required partitions to read/write, and then you can boot the iso because it has somewhere to write data to during boot.

steven-michaud commented 2 years ago

@adespoton, are you saying that unless an external image/drive has a particular partition stored on it, it won't boot on an Apple Silicon machine? And is that partition the "nVME SSD" partition?

And if so, does only the APFS file system have "room" to store this partition ("nVME SSD" or something else)? As opposed to the HFS+J file system?

Edit: Actually, "nVME" seems to be 1) the NVM Express specification and 2) the (physical) logic to implement this specification, which is stored on SSD drives that support NVM Express. If so, I don't see the connection between "nVME" and my test results with my 1TB LaCie external SSD. It presumably does support the NVM Express specification. But it's bootable on both Apple Silicon and Intel hardware if it has a copy of macOS on an APFS partition, and only bootable on Intel hardware if it has a bootable macOS installer on an HFS+J partition.