Open Leoyzen opened 4 years ago
seems adding two patches below we can use newer cpu generation than Penryn with topology defined.But still no luck with odd number cores.
When setting odd cores and topology, the MacOS won't boot, and gives out nothing(after opencore bootloader it just hanged with no panic and no log).Still digging, but already a big progress.
Opencore support is added by https://github.com/acidanthera/OcSupportPkg/pull/17 after release 0.5.2.
When emulating a CPU model (like Skylake-Server in QEMU), the cpu name under system info may be unknown. We can just inject ProcessorType under SMBIOS of config.plist to a specific one, like 0x0F01(3841 in decimal), means 8/10/14/18-cores Xeon-W. Then the cpu name is normal(of course you should use the same core as Xeon-W defination).
It still not work when using 10/14/18-cores topology even with 0x0F01 specified. So it may not related to something Apple Hard check, may be cached related things?(QEMU may not handle cpu cache defination well when using odd cores topology.
Finnally find some clues about this. When booting with 1/3/2(socket/cores/threads) topology,opencore recognized it two ways in log:
00:008 00:004 OCCPU: MP services threads 6 (enabled 6) - Success
00:011 00:003 OCCPU: MP services Pkg 1 Cores 3 Threads 2 - Success
......
00:042 00:003 OCCPU: Pkg 1 Cores 4 Threads 6
So lookup the code, the second one (which is recognized as 4 cores) is using leaf 4 information:
if (Cpu->MaxId >= CPUID_CACHE_PARAMS && (Cpu->Model <= CPU_MODEL_PENRYN || Cpu->Hypervisor)) {
AsmCpuidEx (CPUID_CACHE_PARAMS, 0, &CpuidCacheEax.Uint32, &CpuidCacheEbx.Uint32, NULL, NULL);
if (CpuidCacheEax.Bits.CacheType != CPUID_CACHE_PARAMS_CACHE_TYPE_NULL) {
CoreCount = (UINT16)GetPowerOfTwo32 (CpuidCacheEax.Bits.MaximumAddressableIdsForProcessorCores + 1);
if (CoreCount < CpuidCacheEax.Bits.MaximumAddressableIdsForProcessorCores + 1) {
CoreCount *= 2;
}
Cpu->CoreCount = CoreCount;
//
// We should not be blindly relying on Cpu->Features & CPUID_FEATURE_HTT.
// On Penryn CPUs it is set even without Hyper Threading.
//
if (Cpu->ThreadCount < Cpu->CoreCount) {
Cpu->ThreadCount = Cpu->CoreCount;
}
}
}
So it definately something wrong with cache params setting by QEMU.
Apple have this code too:
reg[eax] = 4; /* cpuid request 4 */
reg[ecx] = index; /* index starting at 0 */
cpuid(reg);
DBG("cpuid(4) index=%d eax=0x%x\n", index, reg[eax]);
cache_type = bitfield32(reg[eax], 4, 0);
if (cache_type == 0) {
break; /* no more caches */
}
cache_level = bitfield32(reg[eax], 7, 5);
cache_sharing = bitfield32(reg[eax], 25, 14) + 1;
info_p->cpuid_cores_per_package
= bitfield32(reg[eax], 31, 26) + 1;
cache_linesize = bitfield32(reg[ebx], 11, 0) + 1;
cache_partitions = bitfield32(reg[ebx], 21, 12) + 1;
cache_associativity = bitfield32(reg[ebx], 31, 22) + 1;
cache_sets = bitfield32(reg[ecx], 31, 0) + 1;
So there are two ways to bypass this logic:
The first one works and the seconds doesn't yet.But we made a big progress now right? We can set odd number cores now.
3-cores/6-threads
<vcpu placement='static' current='6'>8</vcpu>
<vcpus>
<vcpu id='0' enabled='yes' hotpluggable='no' order='1'/>
<vcpu id='1' enabled='yes' hotpluggable='yes' order='2'/>
<vcpu id='2' enabled='yes' hotpluggable='yes' order='3'/>
<vcpu id='3' enabled='yes' hotpluggable='yes' order='4'/>
<vcpu id='4' enabled='yes' hotpluggable='yes' order='5'/>
<vcpu id='5' enabled='yes' hotpluggable='yes' order='6'/>
<vcpu id='6' enabled='no' hotpluggable='yes'/>
<vcpu id='7' enabled='no' hotpluggable='yes'/>
</vcpus>
<cputune>
<vcpupin vcpu='0' cpuset='6'/>
<vcpupin vcpu='1' cpuset='14'/>
<vcpupin vcpu='2' cpuset='7'/>
<vcpupin vcpu='3' cpuset='15'/>
<vcpupin vcpu='4' cpuset='5'/>
<vcpupin vcpu='5' cpuset='13'/>
</cputune>
<cpu mode='host-passthrough' check='none'>
<topology sockets='1' cores='4' threads='2'/>
</cpu>
5-cores/5-threads
<vcpu placement='static' current='5'>8</vcpu>
<vcpus>
<vcpu id='0' enabled='yes' hotpluggable='no' order='1'/>
<vcpu id='1' enabled='yes' hotpluggable='yes' order='2'/>
<vcpu id='2' enabled='yes' hotpluggable='yes' order='3'/>
<vcpu id='3' enabled='yes' hotpluggable='yes' order='4'/>
<vcpu id='4' enabled='yes' hotpluggable='yes' order='5'/>
<vcpu id='5' enabled='no' hotpluggable='yes'/>
<vcpu id='6' enabled='no' hotpluggable='yes'/>
<vcpu id='7' enabled='no' hotpluggable='yes'/>
</vcpus>
<cputune>
<vcpupin vcpu='0' cpuset='6'/>
<vcpupin vcpu='1' cpuset='14'/>
<vcpupin vcpu='2' cpuset='7'/>
<vcpupin vcpu='3' cpuset='15'/>
<vcpupin vcpu='4' cpuset='8'/>
</cputune>
<cpu mode='host-passthrough' check='none'>
<topology sockets='1' cores='8' threads='1'/>
</cpu>
Thanks for sharing the solution. I am not using odd cores, I am passing 8 cores and 2 threads each. With the topology skip patch I was able to have macOS properly recognize the cores/threads, however the CPU name shown is: Single-Core Intel Xeon.
My CPU is an i7-6950X, if I try to passthrough my original CPU with host
in the qemu arguments, the VM gets stuck at Opencore, it doesn't even boot macOS. So I am emulating the Skylake-Server, however, I keep getting the following in the VM logs:
2020-03-30T21:35:05.019712Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.clflushopt [bit 23] 2020-03-30T21:35:05.019715Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.clwb [bit 24] 2020-03-30T21:35:05.019718Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.avx512cd [bit 28] 2020-03-30T21:35:05.019720Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.avx512bw [bit 30] 2020-03-30T21:35:05.019723Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.avx512vl [bit 31] 2020-03-30T21:35:05.019726Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:ECX.pku [bit 3] 2020-03-30T21:35:05.019732Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.0DH:EAX.xsavec [bit 1] 2020-03-30T21:35:05.019736Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.0DH:EAX.xgetbv1 [bit 2] 2020-03-30T21:35:05.019739Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.0DH:EAX [bit 5] 2020-03-30T21:35:05.019742Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.0DH:EAX [bit 6] 2020-03-30T21:35:05.019746Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.0DH:EAX [bit 7] 2020-03-30T21:35:05.019749Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.0DH:EAX [bit 9] 2020-03-30T21:35:05.020568Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.avx512f [bit 16] 2020-03-30T21:35:05.020576Z qemu-system-x86_64: warning: host doesn't support requested feature: CPUID.07H:EBX.avx512dq [bit 17]
@sonoseco This is not error, only warning.Ingore it. passthrough should work.If not, post your opencore log.
@Leoyzen I've emulating the CPU for quite some time, however, I have a lot a audio issues and it seems that can be resolved by passing through the CPU. I still have the same issue as above, where Opencore hangs when using <qemu:arg value='host,vendor=GenuineIntel,+hypervisor,+invtsc,kvm=on,+fma,+avx,+avx2,+aes,+ssse3,+sse4_2,+popcnt,+sse4a,+bmi1,+bmi2'/>
This is the opencore log:
Edit: fixed the issue by using cpu and qemu:arg (adjusted to my config) from this XML: https://github.com/Pavo-IM/Hackinabox/blob/master/Catalina.xml
QEMU patch regarding MSR_CORE_THREAD_COUNT: https://lists.gnu.org/archive/html/qemu-devel/2021-02/msg04906.html
That actually only works for hvf, not kvm. Also see https://lkml.org/lkml/2020/9/16/977
Finnally find some clues about this. When booting with 1/3/2(socket/cores/threads) topology,opencore recognized it two ways in log:
00:008 00:004 OCCPU: MP services threads 6 (enabled 6) - Success 00:011 00:003 OCCPU: MP services Pkg 1 Cores 3 Threads 2 - Success ...... 00:042 00:003 OCCPU: Pkg 1 Cores 4 Threads 6
So lookup the code, the second one (which is recognized as 4 cores) is using leaf 4 information:
if (Cpu->MaxId >= CPUID_CACHE_PARAMS && (Cpu->Model <= CPU_MODEL_PENRYN || Cpu->Hypervisor)) { AsmCpuidEx (CPUID_CACHE_PARAMS, 0, &CpuidCacheEax.Uint32, &CpuidCacheEbx.Uint32, NULL, NULL); if (CpuidCacheEax.Bits.CacheType != CPUID_CACHE_PARAMS_CACHE_TYPE_NULL) { CoreCount = (UINT16)GetPowerOfTwo32 (CpuidCacheEax.Bits.MaximumAddressableIdsForProcessorCores + 1); if (CoreCount < CpuidCacheEax.Bits.MaximumAddressableIdsForProcessorCores + 1) { CoreCount *= 2; } Cpu->CoreCount = CoreCount; // // We should not be blindly relying on Cpu->Features & CPUID_FEATURE_HTT. // On Penryn CPUs it is set even without Hyper Threading. // if (Cpu->ThreadCount < Cpu->CoreCount) { Cpu->ThreadCount = Cpu->CoreCount; } } }
So it definately something wrong with cache params setting by QEMU.
Apple have this code too:
reg[eax] = 4; /* cpuid request 4 */ reg[ecx] = index; /* index starting at 0 */ cpuid(reg); DBG("cpuid(4) index=%d eax=0x%x\n", index, reg[eax]); cache_type = bitfield32(reg[eax], 4, 0); if (cache_type == 0) { break; /* no more caches */ } cache_level = bitfield32(reg[eax], 7, 5); cache_sharing = bitfield32(reg[eax], 25, 14) + 1; info_p->cpuid_cores_per_package = bitfield32(reg[eax], 31, 26) + 1; cache_linesize = bitfield32(reg[ebx], 11, 0) + 1; cache_partitions = bitfield32(reg[ebx], 21, 12) + 1; cache_associativity = bitfield32(reg[ebx], 31, 22) + 1; cache_sets = bitfield32(reg[ecx], 31, 0) + 1;
So there are two ways to bypass this logic:
- setting disable cores in libvirt topology, like 8-cores when 1-core/2-threads disabled
- there are patches in AMD-Vanilla to bypass the logic, like algrey - cpuid_set_cache_info - don't set cpuid_cores_per_package
The first one works and the seconds doesn't yet.But we made a big progress now right? We can set odd number cores now.
@Leoyzen With Proxmox-VE I get the kernel early boot log using 1 sockets 6 cores
================OpenCore 0.7.2 log==================== OC: Watchdog status is 1 00:000 00:000 OC: OpenCore DBG-072-2021-08-02 is loading in Optional mode (0/0)... 00:021 00:021 OC: Boot timestamp - 2021.08.20 06:56:38 00:042 00:021 OCCPU: MP services threads 6 (enabled 6) - Success 00:063 00:020 OCCPU: MP services Pkg 1 Cores 6 Threads 1 - Success 00:083 00:020 OCCPU: Found Intel(R) Core(TM) i7-8700B CPU @ 3.20GHz 00:104 00:021 OCCPU: Signature 906EA Stepping A Model 9E Family 6 Type 0 ExtModel 9 ExtFamily 0 uCode EA 00:126 00:021 OCCPU: Hypervisor detected 00:145 00:019 OCCPU: VMWare TSC: 3191999000Hz, 3191MHz; FSB: 1000000000Hz, 1000MHz 00:165 00:019 OCCPU: EIST CFG Lock 0 00:185 00:019 OCCPU: Detected Apple Processor Type: 07 -> 0709 00:214 00:029 OCCPU: CPUFrequencyFromTSC 0Hz 0MHz 00:234 00:019 OCCPU: CPUFrequency 3191999000Hz 3191MHz 00:254 00:019 OCCPU: FSBFrequency 1000000000Hz 1000MHz 00:275 00:020 OCCPU: Pkg 1 Cores 8 Threads 8 00:294 00:019 OC: OcLoadNvramSupport...
============= kernel log ============= 14:484 00:020 AAPL: #[EB|LOG:EXITBS:START] 2021-08-20T06:56:52 14:505 00:021 AmiShimTimerBoostExit changed current period to 100000
======== End of efiboot serial output. ========
kprintf initialized
Serial mode specified: 00000005
WARNING: Forcing uart driver to output synchronously.printf()s/IOLogs will impact kernel performance.
You are advised to avoid using 'drain_uart_sync' boot-arg.
version_variant = 0
version = Darwin Kernel Version 20.5.0: Sat May 8 05:10:33 PDT 2021; root:xnu-7195.121.3~9/RELEASE_X86_64
KASLR slide: 0x000000000e400000 dynamic
Hiding local relocations
WARNING: ignoring first page in [0x0:0x0]
Physical memory 28672 MB
avail_remaining = 0x6ff5f0
PMAP: PCID enabled
PMAP: Supervisor Mode Execute Protection enabled
PMAP: Supervisor Mode Access Protection enabled
Kernel virtual space from 0xffffff8000000000 to 0xffffffffffffefff.
Available physical space from 0x1301b000 to 0x77ffff000
initialize_screen: b=C0000000, w=00000780, h=00000438, r=00001E00, d=00000001
Darwin Kernel Version 20.5.0: Sat May 8 05:10:33 PDT 2021; root:xnu-7195.121.3~9/RELEASE_X86_64
pmap_startup() delaying init/free of page nums > 0x200000
pmap_startup() init/release time: 29523 microsec
pmap_startup() delayed init/release of 5735343 pages
vm_page_bootstrap: 1461254 free pages, 5846010 wired pages, (up to 5735343 of which are delayed free)
"vm_compressor_mode" is 4
kext submap [0x
Cache Topology Parameters: LLC Depth: 2 Cores Sharing LLC: 6 Threads Sharing LLC: 6 max Sharing of LLC: 8
Logical Topology Parameters: Threads per Core: 1 Cores per Die: 6 Threads per Die: 6 Dies per Package: 1 Cores per Package: 6 Threads per Package: 6
Physical Topology Parameters:
Threads per Core: 1
Cores per Die: 8
Threads per Die: 8
Dies per Package: 0
Cores per Package: 0
Threads per Package: 0
Debugger: Unexpected kernel trap number: 0x0, RIP: 0xffffff800e7af218, CR2: 0x0
CPU 0 panic trap number 0x0, rip 0xffffff800e7af218
cr0 0x0000000080010033 cr2 0x0000000000000000 cr3 0x0000000012ff6000 cr4 0x00000000003200e0
mp_kdp_enter() can't get x86_topo_lock! Debugging anyway! #YOLO
Debugger called:
Backtrace (CPU 0), Frame : Return Address 0xffffff800e59a860 : 0xffffff800e68e0dd mach_kernel : _handle_debugger_trap + 0x3fd 0xffffff800e59a8b0 : 0xffffff800e7d4f33 mach_kernel : _kdp_i386_trap + 0x143 0xffffff800e59a8f0 : 0xffffff800e7c552a mach_kernel : _kernel_trap + 0x55a 0xffffff800e59a940 : 0xffffff800e632a2f mach_kernel : _return_from_trap + 0xff 0xffffff800e59a960 : 0xffffff800e68d8fd mach_kernel : _DebuggerTrapWithState + 0xad 0xffffff800e59aa80 : 0xffffff800e68dbf3 mach_kernel : _panic_trap_to_debugger + 0x273 0xffffff800e59aaf0 : 0xffffff800ee9d81a mach_kernel : _panic + 0x54 0xffffff800e59ab60 : 0xffffff800e7c58f6 mach_kernel : _sync_iss_to_iks + 0x2c6 0xffffff800e59ace0 : 0xffffff800e7c55dd mach_kernel : _kernel_trap + 0x60d 0xffffff800e59ad30 : 0xffffff800e632a2f mach_kernel : _return_from_trap + 0xff 0xffffff800e59ad50 : 0xffffff800e7af218 mach_kernel : _cpu_thread_alloc + 0x6d8 0xffffff800e59ae80 : 0xffffff800e7affd9 mach_kernel : _cpu_thread_init + 0x39 0xffffff800e59aea0 : 0xffffff800e7cec19 mach_kernel : _smp_init + 0x39 0xffffff800e59aec0 : 0xffffff800e6bbd05 mach_kernel : _max_valid_stack_address + 0x435 0xffffff800e59af90 : 0xffffff800e7ca380 mach_kernel : _bootloader_valid_page 0xffffff800e59afb0 : 0xffffff800e7b6913 mach_kernel : _i386_init + 0x443 0xffffff800e59aff0 : 0xffffff800e632158 mach_kernel : _x86_init_wrapper + 0x8 can't perform kext scan: no kext summary Process name corresponding to current thread: Unknown Boot args: keepsyms=1 debug=0x12a msgbuf=1048576 serial=5 -cpuid -topo
Mac OS version: Not yet set
Kernel version: Darwin Kernel Version 20.5.0: Sat May 8 05:10:33 PDT 2021; root:xnu-7195.121.3~9/RELEASE_X86_64 Kernel UUID: 52A1E876-863E-38E3-AC80-09BBAB13B752 KernelCache slide: 0x000000000e400000 KernelCache base: 0xffffff800e600000 Kernel slide: 0x000000000e410000 Kernel text base: 0xffffff800e610000 __HIB text base: 0xffffff800e500000 System shutdown begun: NO Panic diags file unavailable, panic occurred prior to initialization Hibernation exit count: 0
System uptime in nanoseconds: 664522730 Warning: clock is not configured. Can't get time loaded kexts: (none) Attempting to commit panic log to NVRAM
In Memory Panic Stackshot Succeeded Bytes Traced 699 (Uncompressed 720) ** Attempting to commit panic log to NVRAM Attempting system restart... MACH Reboot
Then find the codes in Qemu: https://github.com/qemu/qemu/blob/master/target/i386/cpu.c
case 4: / cache info: needed for Core compatibility / if (cpu->cache_info_passthrough) { host_cpuid(index, count, eax, ebx, ecx, edx); / QEMU gives out its own APIC IDs, never pass down bits 31..26. / eax &= ~0xFC000000; if ((eax & 31) && cs->nr_cores > 1) { eax |= (cs->nr_cores - 1) << 26; } } else if (cpu->vendor_cpuid_only && IS_AMD_CPU(env)) { eax = ebx = ecx = edx = 0; } else { eax = 0; switch (count) { case 0: / L1 dcache info / encode_cache_cpuid4(env->cache_info_cpuid4.l1d_cache, 1, cs->nr_cores, eax, ebx, ecx, edx); break; case 1: / L1 icache info / encode_cache_cpuid4(env->cache_info_cpuid4.l1i_cache, 1, cs->nr_cores, eax, ebx, ecx, edx); break; case 2: / L2 cache info / encode_cache_cpuid4(env->cache_info_cpuid4.l2_cache, cs->nr_threads, cs->nr_cores, eax, ebx, ecx, edx); break; case 3: / L3 cache info / die_offset = apicid_die_offset(&topo_info); if (cpu->enable_l3_cache) { encode_cache_cpuid4(env->cache_info_cpuid4.l3_cache, (1 << die_offset), cs->nr_cores, eax, ebx, ecx, edx); break; } / fall through / default: / end of info / eax = ebx = ecx = edx = 0; break; } } break;
cache_info_passthrough
is false and enable_l3_cache
is true by default, so we can set l3_cache off in Qemu to workaround.
add l3-cache=off
to qemu command line, something like -cpu host,l3-cache=off,kvm=on
Then we run macOS successfully with 6/12 cores but no L3 cache show as flow: `system_profiler SPHardwareDataType Hardware:
Hardware Overview:
Model Name: Mac mini
Model Identifier: Macmini8,1
Processor Name: Unknown
Processor Speed: 3.2 GHz
Number of Processors: 1
Total Number of Cores: 12
L2 Cache (per Core): 4 MB
Memory: 28 GB`
Is setting sockets=3,cores=2 to get 6 cores total not a viable option?
Is setting sockets=3,cores=2 to get 6 cores total not a viable option?
sockets=1,cores=6 may have better performace than sockets=3,cores=2
That actually only works for hvf, not kvm. Also see https://lkml.org/lkml/2020/9/16/977
And now for KVM:
https://lists.nongnu.org/archive/html/qemu-devel/2022-10/msg00579.html
@Toolybird Awesome improvement. Is it usable now?
I've been using AMD-Vanilla for some time but stuck at bigsur because of some unknown (nvme) bugs of msi motherboard.
Maybe it's time to going back kvm-osx again.
If you passthrough your cpu model and defined topology, you will see a kernel panic in MacOS. The problem is divided to two part: Fisrt is bootloader support(see disscusion here). This fixed in latest opencore. The next is Apple MacOS support. Apple use an undocumented Intel msr 0x35(see xen pr) to indentify topology when CPU Model newer than Penryn(Apple get the cpu topology from acpi when using Penryn, that's why Penryn works). The xnu code related:
Of course not only the VM user faces the problem, but also AMD Vanilla users. So we definately can use the way of AMD Vanilla did(using patches), but we need some survey which patches works.