search

DrDonk / unlocker

VMware macOS utilities
MIT License
3.18k stars 442 forks source link

macOS Ventura Testing #47

Closed DrDonk closed 1 year ago

DrDonk commented 1 year ago

It looks like there is a common issue for systems wih AMD and older Intel CPUS. There is no library available for the system.

I will use this thread to track it, but currently do not have an answer to the problem.

Library not loaded: /usr/lib/libSystem.B.dylib

  Referenced from: <2D7AC05B-8AF0-3676-A40A-E40B77ACA459> /sbin/launchd

  Reason: tried: '/usr/lib/libSystem.B.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/lib/libSystem.B.dylib' (no such file), '/usr/lib/libSystem.B.dylib' (no such file, no dyld cache)

AMFI: Denying core dump for pid 1 (launchd)pid 1 exited -- exit reason namespace 6 subcode 0x1, description Library not loaded: /usr/lib/libSystem.B.dylib
  Referenced from: <2D7AC05B-8AF0-3676-A40A-E40B77ACA459> /sbin/launchd
  Reason: tried: '/usr/lib/libSystem.B.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/lib/libSystem.B.dylib' (no such file), '/usr/lib/libSystem.B.dylib' (no such file, no dyld cache)
Debugger called: <panic>

panic(cpu 3 caller 0xffffff800b30a796):  initproc failed to start -- exit reason namespace 6 subcode 0x1 description: Library not loaded: /usr/lib/libSystem.B.dylib
  Referenced from: <2D7AC05B-8AF0-3676-A40A-E40B77ACA459> /sbin/launchd
  Reason: tried: '/usr/lib/libSystem.B.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/usr/lib/libSystem.B.dylib' (no such file), '/usr/lib/libSystem.B.dylib' (no such file, no dyld cache)

uuid info:
 0x10585e000    uuid = <2d7ac05b-8af0-3676-a40a-e40b77aca459>
 0x10f0e4000    uuid = <0f050705-2258-3d40-b7bc-f3b35a44bbea>

Thread 0 crashed

RAX: 0x0000000002000209, RBX: 0x0000000000000000, RCX: 0x00007ff7ba6a0738, RDX: 0x00007ff7ba6a0ba0
RSP: 0x00007ff7ba6a0738, RBP: 0x00007ff7ba6a0780, RSI: 0x0000000000000001, RDI: 0x0000000000000006
R8:  0x00007ff7ba6a07a0, R9:  0x0000000000000000, R10: 0x000000000000003d, R11: 0x0000000000000246
R12: 0x000000000000003d, R13: 0x00007ff7ba6a0ba0, R14: 0x0000000000000001, R15: 0x0000000000000006
RFL: 0x0000000000000246, RIP: 0x000000010f14d83a, CS:  0x0000000000000007, SS:  0x0000000000000023

Thread 0: 0xffffff86e5063598
    0x000000010f14d83a
    0x000000010f1669f9
    0x000000010f0ee1e1
    0x000000010f0eb660
    0x000000010f0ea281
amagerard commented 1 year ago

Debian 11 Bullseye VMware® Workstation Technology Preview 22H2 Pro Unlocker 4.2.3 processor : E5-2697 v2 Motherboard : X79 ram: 32G ECC for server Macosx Ventura makes a loop. I have the same problem . With Windows 10 it's exactly the problem. Maybe my Xeon processor is already too old ! I get the same result with VirtualBox.

It's all good with MacOSX Monterey

DrDonk commented 1 year ago

Ventura will not work if your CPU is pre-Kaby Lake with no AVX2.0. The only possible way forward is to create a form of OCLP patcher for VMware platform. There is no way to patch this in VMware.

Web Site: https://dortania.github.io/OpenCore-Legacy-Patcher/ GitHub Repo: https://github.com/dortania/OpenCore-Legacy-Patcher Ventura details: https://dortania.github.io/OpenCore-Legacy-Patcher/VENTURA-DROP.html

I am not sure how easy this will be, but it would be worth trying for older Intel systems. AMD may well be another matter but I can't comment as not in a position to develop and test for AMD.

DrDonk commented 1 year ago

I have uploaded a utility I wrote to create bootable macOS recovery virtual disks directly from Apple servers. It's uploaded in the wiki page:

https://github.com/DrDonk/unlocker/wiki/Create-a-bootable-macOS-Recovery-vIrtual-disk

My thought process is we could all use the same Ventura test virtual disk that is common to everyone trying to help with the issue. It would allow a simple test of any VMX file changes to be done in a controlled way.

Thoughts?

Anyone up for helping, as I don't have a system that has a problem with Ventura?

ashleyw-gh commented 1 year ago

nice work! Is this intended for VMware desktop products only? Reason I ask is that I generated the vmdk file and uploaded it through to esxi. I was unable to add it to the config of a VM. I also tried to convert the vmdk to a type 4 vmdk for use with esxi;

"C:\Program Files (x86)\VMware\VMware Workstation\vmware-vdiskmanager.exe" -r ventura.vmdk -t 4 ventura-type4.vmdk

this resulted in ventura-type4.vmdk and ventura-type4-flat.vmdk files but I was still unable to add them to the esxi VM.

DrDonk commented 1 year ago

The created VMDK would be a type 0 file which is for the hosted products. That's a limitation of using qemu-img. Let me see what can be done to get it into ESXi.

BTW new version coming as don't need the dmg2img step, qemu-img can do it all.

DrDonk commented 1 year ago

I think vmware-vdiskmanager broke this type 4 stuff some years ago. Seems that the vmkfstools in ESXi can do the job https://kb.vmware.com/s/article/1028042. I've never tried it though.

I'm looking at it but it's lateish in UK so may be tomorrow before I have enough time.

UPDATE:

Try this, upload type-0 file to datastore and run: vmkfstools -i ventura.vmdk -d thin ventura-esxi.vmdk

ashleyw-gh commented 1 year ago

thanks, I tried that but still no go when i try to add it to add the disk to an esxi VM (unsinf esxi8). I've tried a few different variants and tried to recreate the vmdk descriptor etc. I've tried with different virtual hardware levels on the vmx file. Each time when I add the existing disk - it shows the maximum disk file as 0 bytes and will not let me save through the UI.

I think vmware-vdiskmanager broke this type 4 stuff some years ago. Seems that the vmkfstools in ESXi can do the job https://kb.vmware.com/s/article/1028042. I've never tried it though.

I'm looking at it but it's lateish in UK so may be tomorrow before I have enough time.

UPDATE:

Try this, upload type-0 file to datastore and run: vmkfstools -i ventura.vmdk -d thin ventura-esxi.vmdk

ashleyw-gh commented 1 year ago

rather than take this approach I tried the upload from VMware workstation, but it appears the upload to an esxi8 host is currently broken (just my luck eh!). However I tried the latest VMware converter standalone and was able to convert the VM that I created based on your vmdk. Here are the results. On Intel based machine the MacOS installer boot goes through fine and gets to the Language selection option on the install.

I then power off the VM. VMware convertor>convert machine>source: VMware workstation>select mactest>desitnation esxi>convert

I then go to the esxi server. I change the OS type to Mac OS 13 (as the converter defaults to 32bit other). I attempt to power on the VM - the VM hangs at the Apple logo. I power off the VM, then add the folllowing lines into the vmx file;

cpuid.0.eax = "0000:0000:0000:0000:0000:0000:0000:1011"
cpuid.0.ebx = "0111:0101:0110:1110:0110:0101:0100:0111"
cpuid.0.ecx = "0110:1100:0110:0101:0111:0100:0110:1110"
cpuid.0.edx = "0100:1001:0110:0101:0110:1110:0110:1001"
cpuid.1.eax = "0000:0000:0000:0001:0000:0110:0111:0001"
cpuid.1.ebx = "0000:0010:0000:0001:0000:1000:0000:0000"
cpuid.1.ecx = "1000:0010:1001:1000:0010:0010:0000:0011"
cpuid.1.edx = "0000:0111:1000:1011:1111:1011:1111:1111"
smc.version = "0"

then when I power on the VM, it boot loops like before at the Apple logo (with or without a network adaptor). esxi8 host = AMD 5950x VMware workstation: Intel Core i7-4770 CPU

So this is good as I can reproduce the same boot loop issue based on your test on my AMD 5950x VMware server.

DrDonk commented 1 year ago

Good that this can replicate the issue, and thanks for testing. I have a few more checks that I would be grateful for you to try out.

  1. Can you get a monterey recovery disk, and just prove that the the ESXi/Workstation works OK with a recovery disk? It did for me but a double check would be good.
  2. Could you copy the CPUID to the Workstation VMX and see if that impacst both monterey and ventura recovery VMs?

I am also going to concoct some slightly different VMX file settings, and a drop in nvram file to enable debug output without having to boot to the shell and type it in each time.

DrDonk commented 1 year ago

OK quick update it boot loops for me running Monterey or Ventura on a nested ESXi on a real Mac so CPU should be OK.

Using the debug version of VMX causes an assertion:

2022-11-13T12:04:32.027Z Wa(03) vmx - smc: Index 432: Inconsistent read information.
2022-11-13T12:04:32.027Z Cr(01) vmx - PANIC: ASSERT bora/devices/misc/appleSMC.c:599

Checking the binaries and all appears OK, so looks like there is another check that needs patching, at least on ESXi.

It looks like patching SMC is not correct on ESXi8.

ashleyw-gh commented 1 year ago

great debugging! Are you in a position to test the same with a nested ESXi 7u3g hypervisor to see if the boot loop is specific to ESXi8 only?

DrDonk commented 1 year ago

Yes I have set it all up for ESXi7/8 and Monterey/Ventura, and will get on to it ASAP. However got a bad cold so the brain cells aren't firing on all cylinders.

ashleyw-gh commented 1 year ago

On my home esxi8 server, I created a nested esxi7u3g and an esxi8 nested hypervisors, and I tried the Ventura test - it boot loops on both nested esx7 and nested esxi8 hypervisors. The VM on nested esxi7 was set to hw level 19, The VM on nested esxi8 was set to hw level 20.

I then used your utility to generate a test vmdk for Monterey. I repeated the tests on esxi7 and esxi8 (by using vmware converter and taking the running workstation VM across and then adjusting the os ytype to match MacOS. On esxi7u3g the Monterey recovery image goes through to the installer language screen. On esxi8 the Monterey recovery image goes through a boot loop exactly like Ventura.

(note: as my hypervisor tin is an AMD 5950x, I had to apply the CPU masks to the vmx files of the Mac VMs, otherwise the Mac VMs hang on the Apple logo.).

happy to run any other tests - especially as I can run them nested now to prevent having to take down my home server each time.

DrDonk commented 1 year ago

That matches my tests. I will have some more VMX file tests coming but in the meantime could you remove the current cpuid. and the smc.version = 0 settings and try these settings on your real ESXi server please? Actually it would interesting to see what that does in Workstation as well on the Intel CPU.

These settings are from my real MacBook Pro which supports Ventura.

# >>> START <<<
# This spoofs MacBook Pro (15-inch, 2018) - MacBookPro15,1
cpuid.0.ebx = "0111:0101:0110:1110:0110:0101:0100:0111"
cpuid.0.ecx = "0110:1100:0110:0101:0111:0100:0110:1110"
cpuid.0.edx = "0100:1001:0110:0101:0110:1110:0110:1001"
featMask.vm.cpuid.FAMILY = "Val:0x6"
featMask.vm.cpuid.MODEL = "Val:0x9e"
featMask.vm.cpuid.STEPPING = "Val:0x0a"
cpuid.brandstring = "Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz"
cpuid.inhibitDarwinMasks = "TRUE"
board-id.reflectHost = "FALSE"
board-id = "Mac-937A206F2EE63C01"
# >>> END <<<
amagerard commented 1 year ago

I used all your VMDK from Catalina to Ventura on VMware Player and Esxi 7.U3. Everything is good except Ventura. My processor is XEON E5-2697 V2 for VMware Player My processor is XEON E5-2690 for ESXI These 2 processors did not have AVX2.0. Which may explain why Ventura does not start. I did a test with OpenCore Legacy Patcher and I obtained the same bad result only with Ventura. I think I should change my equipment and upgrade to install Esxi8. It's not yet Christmas.

DrDonk commented 1 year ago

I believe I have bad news for ESXi8. The error is something to do with ACPI causing a panic when macOS kernel comes up. Sadly the serial logs are corrupted so cannot pinpoint the reason. However, as you are probably aware VMware have dropped support for macOS as host and guest in ESXi 8 and I'm thinking we're now stuck at ESXi 7.

I am going to concentrate on Ventura on Workstation/Player for now.

For reference, here's the backend of log with panic:

``` SNCoP IL oEcrarlo rV:a rMieatbhloeds paarres ei/neixteicaultiizoend ffaoirl emde t[h\o_dS B[._PSCT0AG]. 2 2N.o_ SATrAg]u m(eNnotdse afrfef fifnfi9t5i3a1l7iez4e5d8 0f)o,r AmEe_tNhOoTd_ F[O_USNTDA ]( 1 6A0C9P3I0 /Eprsrpoarr:s eM-e6t3h2o)d parse/kPEDisableScreen -1 watchdog disabled by boot-arg ACPI BIOS Error (bug): \_SB.PC08._OSC: Object (Alias) must be a control method with 4 argumentsACPI BIOS Error (bug): \_SB.PC0G._OSC: Object (Alias) must be a control method with 4 argumentsexecution fai l(e2d0 1[6\0_9S3B0./PnCs0aGr.gSu3m2eFn.t_sS-T2A2]5) ACPI Warning: (20160930/nsarguments-225) (Node ffffff95317e4580Null Node ptr (20160930/nsobject-383) ACPI W)a,r nAiEn_gN:O T_FOUND (20160930/psparse-632) [ PCI cMatched against SMC (started) Generation from SMC report as 2 Null Node parameter (20160930/nsutils-200) onfiguration begin ] ACPI BIOS Error (bug): \_SB.PC08.De_bOuSgCg:e rA:C PUIn eBxIpOeSc tEerdr okre r(nbeulg )t:r a\p_ SnBu.mPbCer: 0xe, RIP: 0xffffff801697e360, CR2: 00Gx.a_ SC: Previous shutdown cause: 3 ID: Legacy shiCPU 0 panic trap number 0mx e2, rOibpj e0cxtf f(fAflfifa8s0)1 6m9u7set3 6b0e ac rc0o n0tx000000008001003b cr2 0x000000000000000a cr3 0x0000000019ac8000 cr4 0x00000000003606e0 initialize_screen: b=F0000000, w=00000400, h=00000300, r=00001000, d=00000001 Debugger called: panic(cpu 0 caller 0xffffff8015b4aee3): Kernel trap at 0xffffff801697e360, type 14=page fault, registers: CR0: 0x000000008001003b, CR2: 0x000000000000000a, CR3: 0x0000000019ac8000, CR4: 0x00000000003606e0 RAX: 0x0000000000002140, RBX: 0x0000000000000000, RCX: 0x0000000000000009, RDX: 0xffffff80169b47b1 RSP: 0xfffffff2cd7c7bd0, RBP: 0xfffffff2cd7c7c10, RSI: 0xffffff80169af7cd, RDI: 0x00000000000000c9 R8: 0x0000000000000000, R9: 0x0000000000000000, R10: 0x0000000000000008, R11: 0x0000000000000010 R12: 0xffffff80169c0544, R13: 0x0000000000000000, R14: 0x0000000000000000, R15: 0xffffff9064a048c8 RFL: 0x0000000000010296, RIP: 0xffffff801697e360, CS: 0x0000000000000008, SS: 0x0000000000000000 Fault CR2: 0x000000000000000a, Error code: 0x0000000000000000, Fault CPU: 0x0 VMM, PL: 0, VF: 1 Panicked task 0xffffff8b97d27218: 28 threads: pid 0: Backtrace (CPU 0), panicked thread: 0xffffff90648bf0c8, Frame : Return Address 0xfffffff2cd7c75b0 : 0xffffff80159edf9d mach_kernel : _handle_debugger_trap + 0x4ad 0xfffffff2cd7c7600 : 0xffffff8015b5b786 mach_kernel : _kdp_i386_trap + 0x116 0xfffffff2cd7c7640 : 0xffffff8015b4aa10 mach_kernel : _kernel_trap + 0x3e0 0xfffffff2cd7c7690 : 0xffffff8015988951 mach_kernel : _return_from_trap + 0xc1 0xfffffff2cd7c76b0 : 0xffffff80159ee27d mach_kernel : _DebuggerTrapWithState + 0x5d 0xfffffff2cd7c77a0 : 0xffffff80159ed929 mach_kernel : _panic_trap_to_debugger + 0x1a9 0xfffffff2cd7c7800 : 0xffffff80161e0ecb mach_kernel : _panic + 0x84 0xfffffff2cd7c78f0 : 0xffffff8015b4aee3 mach_kernel : _sync_iss_to_iks + 0x2c3 0xfffffff2cd7c7a70 : 0xffffff8015b4abd2 mach_kernel : _kernel_trap + 0x5a2 0xfffffff2cd7c7ac0 : 0xffffff8015988951 mach_kernel : _return_from_trap + 0xc1 0xfffffff2cd7c7ae0 : 0xffffff801697e360 com.apple.driver.AppleACPIPlatform : _AcpiExResolveNodeToValue + 0x11c 0xfffffff2cd7c7c10 : 0xffffff801698659a com.apple.driver.AppleACPIPlatform : _AcpiNsEvaluate + 0x35b 0xfffffff2cd7c7c50 : 0xffffff801698ad46 com.apple.driver.AppleACPIPlatform : _AcpiEvaluateObject + 0x259 0xfffffff2cd7c7cb0 : 0xffffff8016944f2e com.apple.driver.AppleACPIPlatform : _AcpiEvaluateOSCMethod + 0xf7 0xfffffff2cd7c7d60 : 0xffffff801695a30b com.apple.driver.AppleACPIPlatform : __ZN12AppleACPIPCI20enableNativeFeaturesEj + 0x53 0xfffffff2cd7c7d90 : 0xffffff801695a262 com.apple.driver.AppleACPIPlatform : __ZN12AppleACPIPCI5startEP9IOService + 0x100 0xfffffff2cd7c7dd0 : 0xffffff80160e9002 mach_kernel : __ZN9IOService14startCandidateEPS_ + 0xd2 0xfffffff2cd7c7e30 : 0xffffff80160e8b7a mach_kernel : __ZN9IOService15probeCandidatesEP12OSOrderedSet + 0xe0a 0xfffffff2cd7c7ef0 : 0xffffff80160e7be0 mach_kernel : __ZN9IOService14doServiceMatchEj + 0x3b0 0xfffffff2cd7c7f50 : 0xffffff80160eace7 mach_kernel : __ZN15_IOConfigThread4mainEPvi + 0x157 0xfffffff2cd7c7fa0 : 0xffffff801598819e mach_kernel : _call_continuation + 0x2e Kernel Extensions in backtrace: com.apple.driver.AppleACPIPlatform(6.1)[FB7A70DE-6B50-30A3-AD8D-2036437B1A61]@0xffffff801693d000->0xffffff80169b4fff dependency: com.apple.driver.AppleSMC(3.1.9)[712419B7-1229-324C-A182-FA69C7844A2B]@0xffffff801706c000->0xffffff8017088fff dependency: com.apple.iokit.IOACPIFamily(1.4)[55E86E16-9FC5-3768-880B-ADFECF403215]@0xffffff8017f8f000->0xffffff8017f90fff dependency: com.apple.iokit.IOPCIFamily(2.9)[E2CE416B-59DA-3836-9985-FDC527C5E6C7]@0xffffff80183ff000->0xffffff801842efff Process name corresponding to current thread (0xffffff90648bf0c8): Unknown Boot args: -v serial=1 debug=0x108 keepsyms=1 wdt=-1 -no_panic_dialog -topo Mac OS version: Not yet set Kernel version: Darwin Kernel Version 22.1.0: Sun Oct 9 20:14:54 PDT 2022; root:xnu-8792.41.9~2/RELEASE_X86_64 Kernel UUID: BF7C9676-EF23-3E8D-A2E2-25DAC63091B6 roots installed: 0 KernelCache slide: 0x0000000015600000 KernelCache base: 0xffffff8015800000 Kernel slide: 0x00000000156dc000 Kernel text base: 0xffffff80158dc000 __HIB text base: 0xffffff8015700000 System model name: VMware20,1 (Mac-937A206F2EE63C01) System shutdown begun: NO Panic diags file unavailable, panic occurred prior to initialization Hibernation exit count: 0 System uptime in nanoseconds: 16224706247 Last Sleep: absolute base_tsc base_nano Uptime : 0x00000003c8408939 Sleep : 0x0000000000000000 0x0000000000000000 0x0000000000000000 Wake : 0x0000000000000000 0x00000002a3173649 0x0000000000000000 Compressor Info: 0% of compressed pages limit (OK) and 0% of segments limit (OK) with 0 swapfiles and OK swap space Zone info: Zone map: 0xffffff80643da000 - 0xffffffa0643da000 . PGZ : 0xffffff80643da000 - 0xffffff8064bdb000 . VM : 0xffffff8064bdb000 - 0xffffff8531774000 . RO : 0xffffff8531774000 - 0xffffff86cb0a7000 . GEN0 : 0xffffff86cb0a7000 - 0xffffff8b97c40000 . GEN1 : 0xffffff8b97c40000 - 0xffffff90647d9000 . GEN2 : 0xffffff90647d9000 - 0xffffff9531372000 . GEN3 : 0xffffff9531372000 - 0xffffff99fdf0c000 . DATA : 0xffffff99fdf0c000 - 0xffffffa0643da000 Metadata: 0xffffffbdb8f9f000 - 0xffffffbdd8f9f000 Bitmaps : 0xffffffbdd8f9f000 - 0xffffffbdda79f000 last started kext at 15302368695: @private.KextAudit 1.0 (addr 0xffffff80189a4000, size 4096) loaded kexts: @private.KextAudit 1.0 >!AACPIButtons 6.1 >!AHPET 1.8 >!ARTC 2.0.1 >!ASMBIOS 2.1 >!AAPIC 1.7 $!AUserConsent 1 @!ASystemPolicy 2.0.0 @nke.applicationfirewall 403 |IOKitRegistryCompatibility 1 |EndpointSecurity 1 @Dont_Steal_Mac_OS_X 7.0.0 @kec.Compression 1 @kec.!AEncryptedArchive 1 >!AEFINVRAM 2.1 >!AEFIRuntime 2.1 |IOHID!F 2.0.0 |IOTimeSync!F 1110.13 |IOSkywalk!F 1.0 >mDNSOffloadUserClient 1.0.1b8 |IONetworking!F 3.4 >DiskImages 493.0.0 |IO!B!F 9.0.0 |IOReport!F 47 $quarantine 4 $sandbox 300.0 @kext.!AMatch 1.0.0d1 >!ASSE 1.0 >!AKeyStore 2 >!UTDM 547 |IOUSBMass!SDriver 232 |IOSCSIBlockCommandsDevice 476 |IO!S!F 2.1 |IOSCSIArchitectureModel!F 476 >!AMobileFileIntegrity 1.0.5 $!AImage4 5.0.0 @kext.CoreTrust 1 >!AFDEKeyStore 28.30 >!AEffaceable!S 1.0 >!ACredentialManager 1.0 |CoreAnalytics!F 1 >KernelRelayHost 1 |IOUSBHost!F 1.2 >!UHostMergeProperties 1.2 >usb.!UCommon 1.0 >!ABusPower!C 1.0 >!ASEPManager 1.0.1 >IOSlaveProcessor 1 >!AACPIPlatform 6.1 >!ASMC 3.1.9 |IOPCI!F 2.9 |IOACPI!F 1.4 >watchdog 1 @kec.pthread 1 @kec.Libm 1 @kec.corecrypto 12.0 Attempting to commit panic log to NVRAM ** In Memory Panic Stackshot Succeeded ** Bytes Traced 5375 (Uncompressed 12576) ** Attempting to commit panic log to NVRAM Please go to https://panic.apple.com to report this panic ```
DrDonk commented 1 year ago

@amagerard Could you try something for me on VMware Player please? It's not a fix but something I want to check regarding the AVX2.0 instructions.

Can you add the following line to both your Montery and Ventura VMs and then boot? Do you get a message box telling you that AVX is not present on the host?

featMask.vm.cpuid.AVX2 = "Min:1

AdrianEddy commented 1 year ago

I tried adding that line to VMware Player on my AMD Ryzen 5950x, I didn't get any message box about AVX and boot loop is still happening as before

DrDonk commented 1 year ago

Thanks. Ryzen 5950x has AVX2 in which case it would not throw a message. Bottom line is I think we're at the end of the line for some setups due to the new CryptEx system in Ventura.

amagerard commented 1 year ago

The addition of: featMask.vm.cpuid.AVX2 = "Min:1" in the vmx file has the result that I cannot start the virtual machine. I have this message:

Unable to change virtual machine power state: Feature 'cpuid.avx2' was absent, but must be present. Module 'FeatureCompatLate' power on failed. Failed to start the virtual machine.

DrDonk commented 1 year ago

OK. That's good in one way because I can tell everyone to use that setting and it will immediately flag up that their CPU is not compatible with Ventura.

Sorry not a fix and from unlocker perspective there never will be one. Best can hope for is someone works it out using something like Opencore.

Could you attach the vmware.log for the Ventursa VM please? I want to double check the CPUID settings?

cppmyjob commented 1 year ago

Maybe this article https://www.nicksherlock.com/2022/10/installing-macos-13-ventura-on-proxmox will help you understand what is wrong with processors for Venture on Vmware?

jerrywoo96 commented 1 year ago

Based on wikipedia, it requires cpuid.7.ebx bit 5 to be enabled.

But we need to figure out what the rest of the other bits need to be set for cpuid.7.ebx

kenmcd commented 1 year ago

Working link: "https://www.nicksherlock.com/2022/10/installing-macos-13-ventura-on-proxmox/"

jerrywoo96 commented 1 year ago

image

Firstly, we need to upgrade the processor to at least a Haswell, which has a model of 70 (Extended Model ID: 0100, Model ID: 0110). So this will result in: cpuid.1.eax = "0000:0000:0000:0100:0000:0110:0110:1001"

Then enable bit 5 of cpuid.7.ebx. (Which i'm still testing.) Hopefully it works.

jerrywoo96 commented 1 year ago

update: i'm not sure if its me or i'm losing my sanity, can someone double check if setting the 5th bit of cpuid.7.ebx to 1 and upgrading the cpuid to model 70 works for anyone testing? It does not seem to work for me.

How to test: image

1) Copy the circled value and paste it into a hex to bin converter. Eg: 219c0789 should yield a result of 00100001100111000000011110001001

2) Set the 5th bit to 1. Position of bits always starts from the right, index of 0. Eg: 00100001100111000000011110001001 should yeid a result of 00100001100111000000011110101001

3) paste this in the vmx. Eg: cpuid.7.ebx = "0010:0001:1001:1100:0000:0111:1010:1001"

jerrywoo96 commented 1 year ago

I'm not sure what other cpu features haswell has added from ivy bridge that we may need to enable the bits for those features as well. only setting the avx2 bit to 1 also throws the unexpected shutdown error as well.

jerrywoo96 commented 1 year ago

image

TSX consists of HLE and RTM

image LZCNT is in ABM

image FMA3 is just FMA

I've added this in my vmx.

featMask.vm.cpuid.AVX2 = "Min:1"
featMask.vm.cpuid.BMI1 = "Min:1"
featMask.vm.cpuid.BMI2 = "Min:1"
featMask.vm.cpuid.MOVBE = "Min:1"
featMask.vm.cpuid.FMA = "Min:1"
featMask.vm.cpuid.HLE = "Min:1"
featMask.vm.cpuid.RTM = "Min:1"
featMask.vm.cpuid.INVPCID = "Min:1"
featMask.vm.cpuid.ABM = "Min:1"

image

Not sure if macOS uses TSX.

DrDonk commented 1 year ago

REALLY SORRY BUT I ACTUALLY MISSED AN IMPORTANT WORD.

I was actually at a venue with poor lighting when I typed this.

This is NOT going to work. Settng the cpuid bits does NOT give the real CPU the actual AVX 2.0 instructions. VMware virtualises not emulates a CPU.

jerrywoo96 commented 1 year ago

Maybe this article https://www.nicksherlock.com/2022/10/installing-macos-13-ventura-on-proxmox will help you understand what is wrong with processors for Venture on Vmware?

image From: https://github.com/acidanthera/CryptexFixup

i believe all our modern processors should work?

jerrywoo96 commented 1 year ago

This is to going to work. Seeing the cpuid bits does it give the real CPU the actual AVX 2.0 instructions. VMware virtualises not emulates a CPU.

So far what i've tried is still not working. Its 5am now in my region and i've not slept since. I'll experiment again later. nights.

jerrywoo96 commented 1 year ago

Btw i forgot to mention i'm testing it on 5950x using workstation 16.2.4. Using macOS Monterey. If it doesn't work on Monterey, what are the chances it will work on Ventura i thought.

ashleyw-gh commented 1 year ago

@DrDonk , currently is my understanding correct that the only way we are likely to get Ventura working on Workstation/ESXi using AMD (e.g. 5950x) is going to be to use OpenCore https://github.com/AMD-OSX/AMD_Vanilla and then in addition to continue to use unlocker to enable to MACOS X to be selected as an OS type? If that's the case then I guess the only thing that is missing on that is an end to end set of notes that demonstrates that in ESXi and Workstation. If that's the correct path at this stage, then I'll run some tests this week.

DrDonk commented 1 year ago

Thanks for the links on Proxmox but it uses Qemu under the hood in emulation mode so this won't work for virtualised CPU. (I've actually used Qemu to test this out.)

DrDonk commented 1 year ago

@ashleyw-gh I think this may be correct but I'm still not certain. Bottom line is I don't have any AMD kit and this was best endeavours on my part to use my knowledge to help out.

Now I have bare-bones Opencore boot VMDK that I worked on over the summer that can boot macOS with and without unlocker on Intel. I can zip up and post them as a starting point for other to try. I was thinking of starting a new repo with it in. Open to thoughts. I can help with Opencore just not able to test.

DrDonk commented 1 year ago

For now can I suggest if you want to test Ventura on both Intel and CPU you add these settings as defaults.

# >>> START <<<
# This spoofs CPUID as MacBook Pro (15-inch, 2018) - MacBookPro15,1
cpuid.0.ebx = "0111:0101:0110:1110:0110:0101:0100:0111"
cpuid.0.ecx = "0110:1100:0110:0101:0111:0100:0110:1110"
cpuid.0.edx = "0100:1001:0110:0101:0110:1110:0110:1001"
featMask.vm.cpuid.FAMILY = "Val:0x6"
featMask.vm.cpuid.MODEL = "Val:0x9e"
featMask.vm.cpuid.STEPPING = "Val:0x0a"
cpuid.brandstring = "Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz"
cpuid.inhibitDarwinMasks = "TRUE"
board-id.reflectHost = "FALSE"
board-id = "Mac-937A206F2EE63C01"

# Throw VMware error if no AVX2 support in CPU
featMask.vm.cpuid.AVX2 = "Min:1"

# Ensure no boot loop due to macOS unsupported Intel e1000 NIC
ethernet0.virtualDev = "vmxnet3"
# >>> END <<<

It emulates my real MacBook Pro 15,1 for the basic cpuid info & will throw an VMware error on boot if your CPU does not have AVX2, rather than boot loop.

UPDATE: Added Ethernet setting required to stop some boot loops.

ashleyw-gh commented 1 year ago

thanks - I think there is a small typo above; following line is missing a trailing double quote (just in case anyone is copy/pasting)

featMask.vm.cpuid.AVX2 = "Min:1:"

I also saw VMware workstation 17 has just launched and unlocker works as before on Workstation 17.

test1: VMwareWorkstation 17 pro, on Win11 with unlocker, Ventura test on "intel Core i7-4770" CPU, using setttings above. Test gets to region selection. test=success.

test2: Use VMware Convertor standalone and target ESXi 7u3 nested hypervisor on AMD 5950x using settings above. change the OS Type to MacOS 12 (this is the highest MAC OS verison listed for 7u3) after convertor as convertor defaults VM to 32bit other which is incorrect identifier. On power on with settings above, the boot hangs at the apple logo. test=fail

test3: as test2 except try with different combinations of the following enabled in addition to the settings above;

cpuid.0.eax = "0000:0000:0000:0000:0000:0000:0000:1011"
cpuid.1.eax = "0000:0000:0000:0001:0000:0110:0111:0001"
cpuid.1.ebx = "0000:0010:0000:0001:0000:1000:0000:0000"
cpuid.1.ecx = "1000:0010:1001:1000:0010:0010:0000:0011"
cpuid.1.edx = "0000:0111:1000:1011:1111:1011:1111:1111"

either hangs on boot, or "your computer restarted because of a problem". test=fail

DrDonk commented 1 year ago

@ashleyw-gh Thanks for spotting the typo fixed in the post now.

I can confirm WKS 11 on Windows (not tried Linux yet) exhibits same behaviours as WKS 16.

Good tests, thanks. I'm not sure what the converter is doing so will download and test it out as well as building a new VM on ESXi 7 without the converter. Assume you are using the Recovery VMDKs we built earlier.

ashleyw-gh commented 1 year ago

thanks, yes I've been using your Recovery VMDKs and I've run some more tests. test1: Dell PowerEdge R740xd2 - dual socket Intel Xeon Silver 4214R CPU @ 2.40GHz, I created a nested ESXi7 update3g host and installed the unlocker. I used the same VMware convertor trick to take your Ventura recovery image across. I used the recovery image to format a 2nd 80GB thin provisioned drive, and then installed Mac OSx13 onto it without any problems and this runs fine. Test successful.

test2: Dell PowerEdge R740xd2 - dual socket Intel Xeon Silver 4214R CPU @ 2.40GHz, I created a nested ESXi8 host and installed the unlocker. I used the same VMware convertor trick to take your Ventura recovery image across. Starting up the recovery image causes a boot loop. Test fail.

Conclusions;

Some of the confusion could probably be avoided if the unlocker could be modified to correctly work with ESXi8. One of the beauties of unlocker is the ease with which Mac VMs can be spun up. OpenCore on the other hand requires significant time investment, so any documented approach to using OpenCore for both AMD and Intel specifically with VMware workstation/ESXi (assuming that's the only option going forwards) would greatly benefit the community.

My whole motivation for using Mac VMs is for the occasional functional test on my home lab when I need to reproduce an issue across platforms or generate cross platform documentation with minimal equipment and for this Mac VMs are an incredibly useful tool. I've never understood why Apple make it so hard to do this sort of stuff when it actually helps to grow their user base - its the complete opposite to where Linux is at.

DrDonk commented 1 year ago

OK my tests today and success on ESXi 8 on a supported CPU. Read on...

  1. Done on a supported CPU Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
  2. You do need to ensure vmxnet3 is used for NIC (known issue now acknowledged by VMware)
  3. Don't use Workstation/Fusion upload does not create correct VMX file
  4. Don't use Workstation/Fusion connected to ESXi host to create VM on the ESXi host does not create correct VMX file
  5. Don't use VMware Converter also does not work so avoid

Only reliable way is create VM using the ESXi Web UI

ESXi 7 with real SMC: Monterey YES Ventura YES

ESXi 7 no SMC and unlocked: Monterey YES Ventura YES

ESXi 8 with real SMC & ESXi 8 virtual machine: Monterey NO Ventura NO

ESXi 8 with real SMC & ESXi 7.0 U2 virtual machine Monterey YES Ventura YES

ESXi 8 no SMC and unlocked & ESXi 8 virtual machine: Monterey NO Ventura NO

ESXi 8 no SMC and unlocked & ESXi 7.0 U2 virtual machine Monterey YES Ventura YES

Screenshot 2022-11-22 at 18 49 37

The bottom line is macOS support is probably gone from ESXi 8 virtual hardware VMs but it still supports ESXi 7 VMs. This seems to have been confirmed in a roundabout way some VMware KBs and Twitter posts.

I will get back to Workstation soon.

DrDonk commented 1 year ago

When creating a new VM in ESXi 8 UI make sure to select "Compatbility" and select "ESXi 7.0 U2 virtual machine"

Screenshot 2022-11-22 at 20 00 44

Screenshot 2022-11-22 at 20 00 54

ashleyw-gh commented 1 year ago

Awesome, thanks for narrowing that down.

I can confirm on my Dell Intel based server running nested ESXi8, I now have a running Ventura VM I took the following approach;

When I try the exact same process and target an esxi8 host (both bare metal and nested hypervisor) on an AMD 5950x, then the VM starts but hangs on the apple logo.

so I believe the issue is narrowed down specifically to AMD CPUs and Ventura. (and older Intel CPUs not supporting some of the AVX extensions).

I don't have any issues with Ventura and the unlocker on Intel CPU using Workstation 17 (or 16) (provided the Intel CPU is recent enough to support AVX2).

DrDonk commented 1 year ago

Looks like we’re closing in on things. I have idea on a pre-formatted VMDK which has Opencore on it and cryptex fixe from OCLP.

I had previously built minimal Opencore VMDK as there’s a bug in VMware EFI which mean cannot boot to Recovery on an installed VM or change SIP.

I have some other bits to add and my intended outcome is a complete test VM to allow standards tests.

DrDonk commented 1 year ago

I have created 2 test VMs for Workstation/Player. Please download and test on pre-Haswell Intel and AMD CPUs. There is a lot of debugging info and so they are not particularly fast. It should boot to the Ventura Recovery OS.

https://github.com/DrDonk/VMCorePkg/releases/tag/0.0.1

When done can you attach the serial.log and vmware.log files from the VM folder on a post please.

ESXi ones to come.

AdrianEddy commented 1 year ago

AMD 5950X, Windows 11, VMware Workstation 16.2.4 logs.zip

DrDonk commented 1 year ago

@AdrianEddy Many thanks. From the logs neither booted correctly on your setup. This has given a different error regarding CPUID, which may well be different from the pre-Haswell issue and AVX 2.0 instructions.

DrDonk commented 1 year ago

All testers - Another couple of AMD traces from other systems would be useful.

plamen-i commented 1 year ago

Intel(R) Core(TM) i7-2670QM CPU @ 2.20GHz (Sandy Bridge), Windows 10, VMware Workstation 16.2.4 logs.zip

DrDonk commented 1 year ago

@plamen-i @AdrianEddy Could you re-test please with the new 0.0.2 downloads. I made 2 mistakes which may have affected the results.

These are now compatible with ESX and WKS/PLY.

https://github.com/DrDonk/VMCorePkg/releases/tag/0.0.2

DrDonk commented 1 year ago

I'm going to be honest here in that I'm not going to keep this issue open if it cannot be fixed with simple CPUID hacks. Issue tracker is really for fixes to unlocker code and I have always stated that things like AMD support are not really in scope for the unlocker. The same goes for older Intel chips now that Apple has deprecated their usage in Ventura. We all knew this would be coming down the line after the Apple Silicon announcement and VMware's dropping of macOS guests. This means the unlocker will have a definite shelf life.

The bottom line is this would become a duplicate of efforts such as OCLP and other groups who do "Ryzentosh" work. Their solutions should work in a VM, just substitute the VM for the hardware.

I will run these additional tests for now to see if anything can be done, but I'm not confident based on what I've seen so far.