Closed DrDonk closed 1 year ago
Debian 11 Bullseye VMware® Workstation Technology Preview 22H2 Pro Unlocker 4.2.3 processor : E5-2697 v2 Motherboard : X79 ram: 32G ECC for server Macosx Ventura makes a loop. I have the same problem . With Windows 10 it's exactly the problem. Maybe my Xeon processor is already too old ! I get the same result with VirtualBox.
It's all good with MacOSX Monterey
Ventura will not work if your CPU is pre-Kaby Lake with no AVX2.0. The only possible way forward is to create a form of OCLP patcher for VMware platform. There is no way to patch this in VMware.
Web Site: https://dortania.github.io/OpenCore-Legacy-Patcher/ GitHub Repo: https://github.com/dortania/OpenCore-Legacy-Patcher Ventura details: https://dortania.github.io/OpenCore-Legacy-Patcher/VENTURA-DROP.html
I am not sure how easy this will be, but it would be worth trying for older Intel systems. AMD may well be another matter but I can't comment as not in a position to develop and test for AMD.
I have uploaded a utility I wrote to create bootable macOS recovery virtual disks directly from Apple servers. It's uploaded in the wiki page:
https://github.com/DrDonk/unlocker/wiki/Create-a-bootable-macOS-Recovery-vIrtual-disk
My thought process is we could all use the same Ventura test virtual disk that is common to everyone trying to help with the issue. It would allow a simple test of any VMX file changes to be done in a controlled way.
Thoughts?
Anyone up for helping, as I don't have a system that has a problem with Ventura?
nice work! Is this intended for VMware desktop products only? Reason I ask is that I generated the vmdk file and uploaded it through to esxi. I was unable to add it to the config of a VM. I also tried to convert the vmdk to a type 4 vmdk for use with esxi;
"C:\Program Files (x86)\VMware\VMware Workstation\vmware-vdiskmanager.exe" -r ventura.vmdk -t 4 ventura-type4.vmdk
this resulted in ventura-type4.vmdk and ventura-type4-flat.vmdk files but I was still unable to add them to the esxi VM.
The created VMDK would be a type 0 file which is for the hosted products. That's a limitation of using qemu-img. Let me see what can be done to get it into ESXi.
BTW new version coming as don't need the dmg2img step, qemu-img can do it all.
I think vmware-vdiskmanager broke this type 4 stuff some years ago. Seems that the vmkfstools in ESXi can do the job https://kb.vmware.com/s/article/1028042. I've never tried it though.
I'm looking at it but it's lateish in UK so may be tomorrow before I have enough time.
UPDATE:
Try this, upload type-0 file to datastore and run:
vmkfstools -i ventura.vmdk -d thin ventura-esxi.vmdk
thanks, I tried that but still no go when i try to add it to add the disk to an esxi VM (unsinf esxi8). I've tried a few different variants and tried to recreate the vmdk descriptor etc. I've tried with different virtual hardware levels on the vmx file. Each time when I add the existing disk - it shows the maximum disk file as 0 bytes and will not let me save through the UI.
I think vmware-vdiskmanager broke this type 4 stuff some years ago. Seems that the vmkfstools in ESXi can do the job https://kb.vmware.com/s/article/1028042. I've never tried it though.
I'm looking at it but it's lateish in UK so may be tomorrow before I have enough time.
UPDATE:
Try this, upload type-0 file to datastore and run:
vmkfstools -i ventura.vmdk -d thin ventura-esxi.vmdk
rather than take this approach I tried the upload from VMware workstation, but it appears the upload to an esxi8 host is currently broken (just my luck eh!). However I tried the latest VMware converter standalone and was able to convert the VM that I created based on your vmdk. Here are the results. On Intel based machine the MacOS installer boot goes through fine and gets to the Language selection option on the install.
I then power off the VM. VMware convertor>convert machine>source: VMware workstation>select mactest>desitnation esxi>convert
I then go to the esxi server. I change the OS type to Mac OS 13 (as the converter defaults to 32bit other). I attempt to power on the VM - the VM hangs at the Apple logo. I power off the VM, then add the folllowing lines into the vmx file;
cpuid.0.eax = "0000:0000:0000:0000:0000:0000:0000:1011"
cpuid.0.ebx = "0111:0101:0110:1110:0110:0101:0100:0111"
cpuid.0.ecx = "0110:1100:0110:0101:0111:0100:0110:1110"
cpuid.0.edx = "0100:1001:0110:0101:0110:1110:0110:1001"
cpuid.1.eax = "0000:0000:0000:0001:0000:0110:0111:0001"
cpuid.1.ebx = "0000:0010:0000:0001:0000:1000:0000:0000"
cpuid.1.ecx = "1000:0010:1001:1000:0010:0010:0000:0011"
cpuid.1.edx = "0000:0111:1000:1011:1111:1011:1111:1111"
smc.version = "0"
then when I power on the VM, it boot loops like before at the Apple logo (with or without a network adaptor). esxi8 host = AMD 5950x VMware workstation: Intel Core i7-4770 CPU
So this is good as I can reproduce the same boot loop issue based on your test on my AMD 5950x VMware server.
Good that this can replicate the issue, and thanks for testing. I have a few more checks that I would be grateful for you to try out.
I am also going to concoct some slightly different VMX file settings, and a drop in nvram file to enable debug output without having to boot to the shell and type it in each time.
OK quick update it boot loops for me running Monterey or Ventura on a nested ESXi on a real Mac so CPU should be OK.
Using the debug version of VMX causes an assertion:
2022-11-13T12:04:32.027Z Wa(03) vmx - smc: Index 432: Inconsistent read information.
2022-11-13T12:04:32.027Z Cr(01) vmx - PANIC: ASSERT bora/devices/misc/appleSMC.c:599
Checking the binaries and all appears OK, so looks like there is another check that needs patching, at least on ESXi.
It looks like patching SMC is not correct on ESXi8.
great debugging! Are you in a position to test the same with a nested ESXi 7u3g hypervisor to see if the boot loop is specific to ESXi8 only?
Yes I have set it all up for ESXi7/8 and Monterey/Ventura, and will get on to it ASAP. However got a bad cold so the brain cells aren't firing on all cylinders.
On my home esxi8 server, I created a nested esxi7u3g and an esxi8 nested hypervisors, and I tried the Ventura test - it boot loops on both nested esx7 and nested esxi8 hypervisors. The VM on nested esxi7 was set to hw level 19, The VM on nested esxi8 was set to hw level 20.
I then used your utility to generate a test vmdk for Monterey. I repeated the tests on esxi7 and esxi8 (by using vmware converter and taking the running workstation VM across and then adjusting the os ytype to match MacOS. On esxi7u3g the Monterey recovery image goes through to the installer language screen. On esxi8 the Monterey recovery image goes through a boot loop exactly like Ventura.
(note: as my hypervisor tin is an AMD 5950x, I had to apply the CPU masks to the vmx files of the Mac VMs, otherwise the Mac VMs hang on the Apple logo.).
happy to run any other tests - especially as I can run them nested now to prevent having to take down my home server each time.
That matches my tests. I will have some more VMX file tests coming but in the meantime could you remove the current cpuid.
These settings are from my real MacBook Pro which supports Ventura.
# >>> START <<<
# This spoofs MacBook Pro (15-inch, 2018) - MacBookPro15,1
cpuid.0.ebx = "0111:0101:0110:1110:0110:0101:0100:0111"
cpuid.0.ecx = "0110:1100:0110:0101:0111:0100:0110:1110"
cpuid.0.edx = "0100:1001:0110:0101:0110:1110:0110:1001"
featMask.vm.cpuid.FAMILY = "Val:0x6"
featMask.vm.cpuid.MODEL = "Val:0x9e"
featMask.vm.cpuid.STEPPING = "Val:0x0a"
cpuid.brandstring = "Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz"
cpuid.inhibitDarwinMasks = "TRUE"
board-id.reflectHost = "FALSE"
board-id = "Mac-937A206F2EE63C01"
# >>> END <<<
I used all your VMDK from Catalina to Ventura on VMware Player and Esxi 7.U3. Everything is good except Ventura. My processor is XEON E5-2697 V2 for VMware Player My processor is XEON E5-2690 for ESXI These 2 processors did not have AVX2.0. Which may explain why Ventura does not start. I did a test with OpenCore Legacy Patcher and I obtained the same bad result only with Ventura. I think I should change my equipment and upgrade to install Esxi8. It's not yet Christmas.
I believe I have bad news for ESXi8. The error is something to do with ACPI causing a panic when macOS kernel comes up. Sadly the serial logs are corrupted so cannot pinpoint the reason. However, as you are probably aware VMware have dropped support for macOS as host and guest in ESXi 8 and I'm thinking we're now stuck at ESXi 7.
I am going to concentrate on Ventura on Workstation/Player for now.
For reference, here's the backend of log with panic:
@amagerard Could you try something for me on VMware Player please? It's not a fix but something I want to check regarding the AVX2.0 instructions.
Can you add the following line to both your Montery and Ventura VMs and then boot? Do you get a message box telling you that AVX is not present on the host?
featMask.vm.cpuid.AVX2 = "Min:1
I tried adding that line to VMware Player on my AMD Ryzen 5950x, I didn't get any message box about AVX and boot loop is still happening as before
Thanks. Ryzen 5950x has AVX2 in which case it would not throw a message. Bottom line is I think we're at the end of the line for some setups due to the new CryptEx system in Ventura.
The addition of: featMask.vm.cpuid.AVX2 = "Min:1" in the vmx file has the result that I cannot start the virtual machine. I have this message:
Unable to change virtual machine power state: Feature 'cpuid.avx2' was absent, but must be present. Module 'FeatureCompatLate' power on failed. Failed to start the virtual machine.
OK. That's good in one way because I can tell everyone to use that setting and it will immediately flag up that their CPU is not compatible with Ventura.
Sorry not a fix and from unlocker perspective there never will be one. Best can hope for is someone works it out using something like Opencore.
Could you attach the vmware.log for the Ventursa VM please? I want to double check the CPUID settings?
Maybe this article https://www.nicksherlock.com/2022/10/installing-macos-13-ventura-on-proxmox will help you understand what is wrong with processors for Venture on Vmware?
Based on wikipedia, it requires cpuid.7.ebx bit 5 to be enabled.
But we need to figure out what the rest of the other bits need to be set for cpuid.7.ebx
Firstly, we need to upgrade the processor to at least a Haswell, which has a model of 70 (Extended Model ID: 0100, Model ID: 0110). So this will result in: cpuid.1.eax = "0000:0000:0000:0100:0000:0110:0110:1001"
Then enable bit 5 of cpuid.7.ebx. (Which i'm still testing.) Hopefully it works.
update: i'm not sure if its me or i'm losing my sanity, can someone double check if setting the 5th bit of cpuid.7.ebx to 1 and upgrading the cpuid to model 70 works for anyone testing? It does not seem to work for me.
How to test:
1) Copy the circled value and paste it into a hex to bin converter. Eg: 219c0789
should yield a result of 00100001100111000000011110001001
2) Set the 5th bit to 1. Position of bits always starts from the right, index of 0. Eg: 00100001100111000000011110001001
should yeid a result of 00100001100111000000011110101001
3) paste this in the vmx. Eg: cpuid.7.ebx = "0010:0001:1001:1100:0000:0111:1010:1001"
I'm not sure what other cpu features haswell has added from ivy bridge that we may need to enable the bits for those features as well. only setting the avx2 bit to 1 also throws the unexpected shutdown error as well.
LZCNT is in ABM
FMA3 is just FMA
I've added this in my vmx.
featMask.vm.cpuid.AVX2 = "Min:1"
featMask.vm.cpuid.BMI1 = "Min:1"
featMask.vm.cpuid.BMI2 = "Min:1"
featMask.vm.cpuid.MOVBE = "Min:1"
featMask.vm.cpuid.FMA = "Min:1"
featMask.vm.cpuid.HLE = "Min:1"
featMask.vm.cpuid.RTM = "Min:1"
featMask.vm.cpuid.INVPCID = "Min:1"
featMask.vm.cpuid.ABM = "Min:1"
Not sure if macOS uses TSX.
REALLY SORRY BUT I ACTUALLY MISSED AN IMPORTANT WORD.
I was actually at a venue with poor lighting when I typed this.
This is NOT going to work. Settng the cpuid bits does NOT give the real CPU the actual AVX 2.0 instructions. VMware virtualises not emulates a CPU.
Maybe this article https://www.nicksherlock.com/2022/10/installing-macos-13-ventura-on-proxmox will help you understand what is wrong with processors for Venture on Vmware?
From: https://github.com/acidanthera/CryptexFixup
i believe all our modern processors should work?
This is to going to work. Seeing the cpuid bits does it give the real CPU the actual AVX 2.0 instructions. VMware virtualises not emulates a CPU.
So far what i've tried is still not working. Its 5am now in my region and i've not slept since. I'll experiment again later. nights.
Btw i forgot to mention i'm testing it on 5950x using workstation 16.2.4. Using macOS Monterey. If it doesn't work on Monterey, what are the chances it will work on Ventura i thought.
@DrDonk , currently is my understanding correct that the only way we are likely to get Ventura working on Workstation/ESXi using AMD (e.g. 5950x) is going to be to use OpenCore https://github.com/AMD-OSX/AMD_Vanilla and then in addition to continue to use unlocker to enable to MACOS X to be selected as an OS type? If that's the case then I guess the only thing that is missing on that is an end to end set of notes that demonstrates that in ESXi and Workstation. If that's the correct path at this stage, then I'll run some tests this week.
Thanks for the links on Proxmox but it uses Qemu under the hood in emulation mode so this won't work for virtualised CPU. (I've actually used Qemu to test this out.)
@ashleyw-gh I think this may be correct but I'm still not certain. Bottom line is I don't have any AMD kit and this was best endeavours on my part to use my knowledge to help out.
Now I have bare-bones Opencore boot VMDK that I worked on over the summer that can boot macOS with and without unlocker on Intel. I can zip up and post them as a starting point for other to try. I was thinking of starting a new repo with it in. Open to thoughts. I can help with Opencore just not able to test.
For now can I suggest if you want to test Ventura on both Intel and CPU you add these settings as defaults.
# >>> START <<<
# This spoofs CPUID as MacBook Pro (15-inch, 2018) - MacBookPro15,1
cpuid.0.ebx = "0111:0101:0110:1110:0110:0101:0100:0111"
cpuid.0.ecx = "0110:1100:0110:0101:0111:0100:0110:1110"
cpuid.0.edx = "0100:1001:0110:0101:0110:1110:0110:1001"
featMask.vm.cpuid.FAMILY = "Val:0x6"
featMask.vm.cpuid.MODEL = "Val:0x9e"
featMask.vm.cpuid.STEPPING = "Val:0x0a"
cpuid.brandstring = "Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz"
cpuid.inhibitDarwinMasks = "TRUE"
board-id.reflectHost = "FALSE"
board-id = "Mac-937A206F2EE63C01"
# Throw VMware error if no AVX2 support in CPU
featMask.vm.cpuid.AVX2 = "Min:1"
# Ensure no boot loop due to macOS unsupported Intel e1000 NIC
ethernet0.virtualDev = "vmxnet3"
# >>> END <<<
It emulates my real MacBook Pro 15,1 for the basic cpuid info & will throw an VMware error on boot if your CPU does not have AVX2, rather than boot loop.
UPDATE: Added Ethernet setting required to stop some boot loops.
thanks - I think there is a small typo above; following line is missing a trailing double quote (just in case anyone is copy/pasting)
featMask.vm.cpuid.AVX2 = "Min:1:"
I also saw VMware workstation 17 has just launched and unlocker works as before on Workstation 17.
test1: VMwareWorkstation 17 pro, on Win11 with unlocker, Ventura test on "intel Core i7-4770" CPU, using setttings above. Test gets to region selection. test=success.
test2: Use VMware Convertor standalone and target ESXi 7u3 nested hypervisor on AMD 5950x using settings above. change the OS Type to MacOS 12 (this is the highest MAC OS verison listed for 7u3) after convertor as convertor defaults VM to 32bit other which is incorrect identifier. On power on with settings above, the boot hangs at the apple logo. test=fail
test3: as test2 except try with different combinations of the following enabled in addition to the settings above;
cpuid.0.eax = "0000:0000:0000:0000:0000:0000:0000:1011"
cpuid.1.eax = "0000:0000:0000:0001:0000:0110:0111:0001"
cpuid.1.ebx = "0000:0010:0000:0001:0000:1000:0000:0000"
cpuid.1.ecx = "1000:0010:1001:1000:0010:0010:0000:0011"
cpuid.1.edx = "0000:0111:1000:1011:1111:1011:1111:1111"
either hangs on boot, or "your computer restarted because of a problem". test=fail
@ashleyw-gh Thanks for spotting the typo fixed in the post now.
I can confirm WKS 11 on Windows (not tried Linux yet) exhibits same behaviours as WKS 16.
Good tests, thanks. I'm not sure what the converter is doing so will download and test it out as well as building a new VM on ESXi 7 without the converter. Assume you are using the Recovery VMDKs we built earlier.
thanks, yes I've been using your Recovery VMDKs and I've run some more tests. test1: Dell PowerEdge R740xd2 - dual socket Intel Xeon Silver 4214R CPU @ 2.40GHz, I created a nested ESXi7 update3g host and installed the unlocker. I used the same VMware convertor trick to take your Ventura recovery image across. I used the recovery image to format a 2nd 80GB thin provisioned drive, and then installed Mac OSx13 onto it without any problems and this runs fine. Test successful.
test2: Dell PowerEdge R740xd2 - dual socket Intel Xeon Silver 4214R CPU @ 2.40GHz, I created a nested ESXi8 host and installed the unlocker. I used the same VMware convertor trick to take your Ventura recovery image across. Starting up the recovery image causes a boot loop. Test fail.
Conclusions;
Some of the confusion could probably be avoided if the unlocker could be modified to correctly work with ESXi8. One of the beauties of unlocker is the ease with which Mac VMs can be spun up. OpenCore on the other hand requires significant time investment, so any documented approach to using OpenCore for both AMD and Intel specifically with VMware workstation/ESXi (assuming that's the only option going forwards) would greatly benefit the community.
My whole motivation for using Mac VMs is for the occasional functional test on my home lab when I need to reproduce an issue across platforms or generate cross platform documentation with minimal equipment and for this Mac VMs are an incredibly useful tool. I've never understood why Apple make it so hard to do this sort of stuff when it actually helps to grow their user base - its the complete opposite to where Linux is at.
OK my tests today and success on ESXi 8 on a supported CPU. Read on...
Only reliable way is create VM using the ESXi Web UI
ESXi 7 with real SMC: Monterey YES Ventura YES
ESXi 7 no SMC and unlocked: Monterey YES Ventura YES
ESXi 8 with real SMC & ESXi 8 virtual machine: Monterey NO Ventura NO
ESXi 8 with real SMC & ESXi 7.0 U2 virtual machine Monterey YES Ventura YES
ESXi 8 no SMC and unlocked & ESXi 8 virtual machine: Monterey NO Ventura NO
ESXi 8 no SMC and unlocked & ESXi 7.0 U2 virtual machine Monterey YES Ventura YES
The bottom line is macOS support is probably gone from ESXi 8 virtual hardware VMs but it still supports ESXi 7 VMs. This seems to have been confirmed in a roundabout way some VMware KBs and Twitter posts.
I will get back to Workstation soon.
When creating a new VM in ESXi 8 UI make sure to select "Compatbility" and select "ESXi 7.0 U2 virtual machine"
Awesome, thanks for narrowing that down.
I can confirm on my Dell Intel based server running nested ESXi8, I now have a running Ventura VM I took the following approach;
When I try the exact same process and target an esxi8 host (both bare metal and nested hypervisor) on an AMD 5950x, then the VM starts but hangs on the apple logo.
so I believe the issue is narrowed down specifically to AMD CPUs and Ventura. (and older Intel CPUs not supporting some of the AVX extensions).
I don't have any issues with Ventura and the unlocker on Intel CPU using Workstation 17 (or 16) (provided the Intel CPU is recent enough to support AVX2).
Looks like we’re closing in on things. I have idea on a pre-formatted VMDK which has Opencore on it and cryptex fixe from OCLP.
I had previously built minimal Opencore VMDK as there’s a bug in VMware EFI which mean cannot boot to Recovery on an installed VM or change SIP.
I have some other bits to add and my intended outcome is a complete test VM to allow standards tests.
I have created 2 test VMs for Workstation/Player. Please download and test on pre-Haswell Intel and AMD CPUs. There is a lot of debugging info and so they are not particularly fast. It should boot to the Ventura Recovery OS.
https://github.com/DrDonk/VMCorePkg/releases/tag/0.0.1
When done can you attach the serial.log and vmware.log files from the VM folder on a post please.
ESXi ones to come.
AMD 5950X, Windows 11, VMware Workstation 16.2.4 logs.zip
@AdrianEddy Many thanks. From the logs neither booted correctly on your setup. This has given a different error regarding CPUID, which may well be different from the pre-Haswell issue and AVX 2.0 instructions.
All testers - Another couple of AMD traces from other systems would be useful.
Intel(R) Core(TM) i7-2670QM CPU @ 2.20GHz (Sandy Bridge), Windows 10, VMware Workstation 16.2.4 logs.zip
@plamen-i @AdrianEddy Could you re-test please with the new 0.0.2 downloads. I made 2 mistakes which may have affected the results.
These are now compatible with ESX and WKS/PLY.
I'm going to be honest here in that I'm not going to keep this issue open if it cannot be fixed with simple CPUID hacks. Issue tracker is really for fixes to unlocker code and I have always stated that things like AMD support are not really in scope for the unlocker. The same goes for older Intel chips now that Apple has deprecated their usage in Ventura. We all knew this would be coming down the line after the Apple Silicon announcement and VMware's dropping of macOS guests. This means the unlocker will have a definite shelf life.
The bottom line is this would become a duplicate of efforts such as OCLP and other groups who do "Ryzentosh" work. Their solutions should work in a VM, just substitute the VM for the hardware.
I will run these additional tests for now to see if anything can be done, but I'm not confident based on what I've seen so far.
It looks like there is a common issue for systems wih AMD and older Intel CPUS. There is no library available for the system.
I will use this thread to track it, but currently do not have an answer to the problem.