[Solved] crash on Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz

[chenkaigithub]

Crash when insmod ./corefreqk.ko

CPU :  Intel(R) Xeon(R) Gold 6126 CPU @ 2.60GHz
OS centos 7 with kernel 5.2.11

[vmcore-dmesg.txt](https://github.com/cyring/CoreFreq/files/7231612/vmcore-dmesg.txt)

[245.004245] corefreqk: loading out-of-tree module taints kernel.
[  245.330388] CoreFreq(4:28): Processor [ 06_55] Architecture [Skylake/X] SMT [48/48]
[  245.330447] general protection fault: 0000 [#1] SMP PTI
[  245.330452] CPU: 4 PID: 0 Comm: swapper/4 Kdump: loaded Tainted: G           O      5.2.11 #1
[  245.330454] Hardware name: Dell Inc. VxRail P570F/0923K0, BIOS 2.12.2 07/09/2021
[  245.330475] RIP: 0010:Intel_DomainPowerLimit+0x18/0x230 [corefreqk]
[  245.330478] Code: c1 e2 20 89 c0 48 09 c2 48 89 96 20 11 00 00 c3 66 90 0f 1f 44 00 00 41 55 41 89 d3 89 f9 47 8d 0c 1b 41 54 55 41 8d 69 01 53 <0f> 32 49 89 d0 89 c0 31 db 49 c1 e0 20 49 09 c0 48 8b 05 f9 3f 03
[  245.330480] RSP: 0018:ffffa5b146908f08 EFLAGS: 00010046
[  245.330482] RAX: 0000000000000000 RBX: ffff96eddcea4000 RCX: 000000000000065c
[  245.330483] RDX: 0000000000000004 RSI: 8000000000000000 RDI: 000000000000065c
[  245.330484] RBP: 0000000000000009 R08: 0000000000000000 R09: 0000000000000008
[  245.330486] R10: ffff96e6836ba000 R11: 0000000000000004 R12: ffffa5b146908f40
[  245.330487] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[  245.330489] FS:  0000000000000000(0000) GS:ffff96eddfa80000(0000) knlGS:0000000000000000
[  245.330490] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  245.330492] CR2: 00007fe3e1fd35b0 CR3: 000000015da0a005 CR4: 00000000007606e0
[  245.330493] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  245.330494] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[  245.330495] PKRU: 55555554
[  245.330496] Call Trace:
[  245.330500]  <IRQ>
[  245.330512]  PerCore_Skylake_X_Query+0x275/0x2b0 [corefreqk]
[  245.330530]  ? Intel_Core_Counters_Set+0x100/0x100 [corefreqk]
[  245.330539]  Start_Skylake_X+0x62/0x2c0 [corefreqk]
[  245.330547]  ? Cycle_Skylake+0x7b0/0x7b0 [corefreqk]
[  245.330555]  ? Intel_Core_Counters_Set+0x100/0x100 [corefreqk]
[  245.330561]  flush_smp_call_function_queue+0x45/0x120
[  245.330566]  smp_call_function_single_interrupt+0x3a/0xd0
[  245.330569]  call_function_single_interrupt+0xf/0x20
[  245.330571]  </IRQ>
[  245.330575] RIP: 0010:cpuidle_enter_state+0xba/0x410
[  245.330577] Code: e8 5b 16 a5 ff 44 8b 63 04 49 89 c6 0f 1f 44 00 00 31 ff e8 38 27 a5 ff 80 7c 24 0b 00 0f 85 23 02 00 00 fb 66 0f 1f 44 00 00 <45> 85 ed 0f 88 32 02 00 00 49 63 f5 4c 2b 34 24 48 ba cf f7 53 e3
[  245.330579] RSP: 0018:ffffa5b1466f7e70 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff04
[  245.330581] RAX: ffff96eddfaaaf80 RBX: ffffc5a93fc84678 RCX: 000000000000001f
[  245.330582] RDX: 0000000000000000 RSI: 0000000000000002 RDI: 0000000000000000
[  245.330583] RBP: ffffffff82334460 R08: 0000000000000002 R09: fff8b5d44d13f7d4
[  245.330585] R10: 0000000000000018 R11: 071c71c71c71c71c R12: 0000000000000004
[  245.330586] R13: 0000000000000003 R14: 000000391ed4d8db R15: 0000000000000004
[  245.330590]  cpuidle_enter+0x29/0x40
[  245.330599]  do_idle+0x1ed/0x290
[  245.330603]  cpu_startup_entry+0x19/0x20
[  245.330607]  start_secondary+0x180/0x1d0
[  245.330612]  secondary_startup_64+0xa4/0xb0
[  245.330614] Modules linked in: corefreqk(O) xt_CHECKSUM xt_MASQUERADE tun bridge stp llc ip6t_rpfilter ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_conntrack ebtable_nat ebtable_broute ip6table_nat ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_nat iptable_mangle iptable_security iptable_raw nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip_set nfnetlink ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ib_isert iscsi_target_mod ib_srpt target_core_mod ib_srp scsi_transport_srp rpcrdma sunrpc rdma_ucm ib_iser ib_umad intel_rapl skx_edac nfit libnvdimm x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel rdma_cm iw_cm vfat fat ib_ipoib kvm libiscsi ipmi_ssif ib_cm scsi_transport_iscsi irqbypass mlx5_ib crct10dif_pclmul crc32_pclmul iTCO_wdt ghash_clmulni_intel iTCO_vendor_support ib_uverbs dell_smbios dell_wmi_descriptor wmi_bmof ib_core dcdbas aesni_intel crypto_simd ses cryptd glue_helper pcspkr enclosure cdc_ether usbnet joydev mei_me mii sg
[  245.330656]  i2c_i801 lpc_ich mei ioatdma dca ipmi_si wmi ipmi_devintf pcc_cpufreq ipmi_msghandler acpi_power_meter ip_tables xfs libcrc32c sd_mod mgag200 i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt crc32c_intel fb_sys_fops mlx5_core nvme ahci ttm mpt3sas mlxfw libahci nvme_core ptp raid_class drm pps_core scsi_transport_sas libata uas usb_storage dm_mirror dm_region_hash dm_log dm_mod fuse

[cyring]

One of these Intel_DomainPowerLimit() calls don't work with your Skylake/X

• From the develop branch, can you comment out the whole statement at: https://github.com/cyring/CoreFreq/blob/6e1d7d414017a24cce69edb24c5586324d0c1fa4/corefreqk.c#L9661

• with:

  if (Core->Bind == PUBLIC(RO(Proc))->Service.Core) {
  /*
      Intel_DomainPowerLimit( MSR_PKG_POWER_LIMIT,
                  PKG_POWER_LIMIT_LOCK_MASK,
                  PWR_DOMAIN(PKG) );
  
      Intel_DomainPowerLimit( MSR_PP0_POWER_LIMIT,
                  PPn_POWER_LIMIT_LOCK_MASK,
                  PWR_DOMAIN(CORES) );
  
      Intel_DomainPowerLimit( MSR_PLATFORM_POWER_LIMIT,
                  PKG_POWER_LIMIT_LOCK_MASK,
                  PWR_DOMAIN(PLATFORM) );
  
      Intel_DomainPowerLimit( MSR_DRAM_POWER_LIMIT,
                  PPn_POWER_LIMIT_LOCK_MASK,
                  PWR_DOMAIN(RAM) );
  */
      Intel_Watchdog(Core);
  }

• Next rebuild all with make clean all

• Make sure Daemon and Driver are fully removed

• Now test the built Driver

If things get better then un-comment one after the other each Intel_DomainPowerLimit() until you encounter the crash again.

You should then be able to tell me which among MSR_DRAM_POWER_LIMIT, MSR_PLATFORM_POWER_LIMIT, MSR_PP0_POWER_LIMIT and MSR_PKG_POWER_LIMIT is forbidden with your processor ?

EDIT: looking at your RCX dump, MSR_PLATFORM_POWER_LIMIT is the faulty one: https://github.com/cyring/CoreFreq/blob/6e1d7d414017a24cce69edb24c5586324d0c1fa4/intelmsr.h#L207 I think you can just comment out its call.

[chenkaigithub]

@cyring Thank you for your quick response. I follow your instruction.

MSR_PLATFORM_POWER_LIMIT and MSR_DRAM_POWER_LIMIT each will cause system crash. Please see the attachment.

vmcore-dmesg(MSR_DRAM_POWER_LIMIT).txt vmcore-dmesg(MSR_PLATFORM_POWER_LIMIT).txt

[cyring]

@cyring Thank you for your quick response. I follow your instruction.

MSR_PLATFORM_POWER_LIMIT and MSR_DRAM_POWER_LIMIT each will cause system crash. Please see the attachment.

vmcore-dmesg(MSR_DRAM_POWER_LIMIT).txt vmcore-dmesg(MSR_PLATFORM_POWER_LIMIT).txt

Well done!

Can you now fetch the develop branch where is supplied the fix and provide various outputs of CoreFreq like one dedicated page "Xeon Gold 6126" in the Wiki support ? This page as an example. The output of corefreq-cli -s will especially help me to chase any remaining bugs or mistakes.

Thank you.

[chenkaigithub]

Thank you for your quick fix. The wiki is updated.

[cyring]

Thank you for your quick fix. The wiki is updated.

Thank you for these additions.

First time I can see PPIN# showing up !

Many other small things to fix or investiguate:

• Turbo Boost starting at 9C
• Turbo AUTO < 255 > is odd but it may come from hardware The side effect is that the UI ruler is scaled to 255
• I/O MMU Version to query
• Lowest C-State LIMIT < C0> and Max C-State Inclusion RANGE < C0> are both w/o C-states

[chenkaigithub]

I am not an expert on the CPU part. I can help you to verify something.

[cyring]

I am not an expert on the CPU part. I can help you to verify something.

I have first to enhance code facing data provided by your Skylake/X. For example, make the UI based on validated ratios; create workarounds of garbage or uninitialized values.

I'll be back with additional testing requests. Thanks for your help. CyrIng

[cyring]

Among latest commits in develop branch, the UI ruler is now limited to a Max Ratio which is computed from the Makefile directive MAX_FREQ_HZ and the Base Clock

Please give it a try with Turbo boosted single Core. You can use the integrated tools to stress CPU: Press O and Turbo Round Robin to stress CPU one after one. Accordingly to your screenshot, I expect the drawing to be limited to 37, your highest boosted ratio in 1C

[chenkaigithub]

I will check this after Oct 8 for now on holidays.

[cyring]

I will check this after Oct 8 for now on holidays.

Feel free to come back. Regards

[chenkaigithub]

Sorry. I am busy these days. Dev branch can not be compiled.

[cyring]

Sorry. I am busy these days. Dev branch can not be compiled.

I was too ambitious to use __auto_type as saint grale It is deleted in the last develop branch commit.

[chenkaigithub]

It could be compiled but crash. vmcore-dmesg.txt .

[cyring]

It could be compiled but crash. vmcore-dmesg.txt .

Fix is available in develop branch although I wonder if during our previous tests, we failed while probing the TCO register ?

[cyring]

EDIT: before pulling the latest commit ff41125ef7a6375668bb45b994125ae2b270dfa8, can you retry but this time, unloading or black-listing the TCO drivers iTCO_wdt and iTCO_vendor_support (and module dependencies if needed), prior starting the corefreqk.ko module.

The IRQ trapped in the Call trace of your kernel dump is a hint that those drivers may have installed an interrupt handler which conflicts with CoreFreq driver.

[chenkaigithub]

EDIT: before pulling the latest commit ff41125, can you retry but this time, unloading or black-listing the TCO drivers iTCO_wdt and iTCO_vendor_support (and module dependencies if needed), prior starting the corefreqk.ko module.

The IRQ trapped in the Call trace of your kernel dump is a hint that those drivers may have installed an interrupt handler which conflicts with CoreFreq driver.

CyrIng, How unloading or black-listing the TCO drivers?

[cyring]

EDIT: before pulling the latest commit ff41125, can you retry but this time, unloading or black-listing the TCO drivers iTCO_wdt and iTCO_vendor_support (and module dependencies if needed), prior starting the corefreqk.ko module. The IRQ trapped in the Call trace of your kernel dump is a hint that those drivers may have installed an interrupt handler which conflicts with CoreFreq driver.

CyrIng, How unloading or black-listing the TCO drivers?

• Unloading modules: modprobe -r or rmmod followed by the module name

• Blacklisting modules: This prevents drivers to be auto loaded during boot. When you can't unload a module because it is forced resident by kernel, blacklisting is your choice. This is a boot command line argument you will set in your boot loader configuration:

• grub.cfg

• syslinux.cfg

• /EFI/loader/...

Here my arguments for Xeon W3690 This a bare-metal model where CoreFreq will be the kernel governor (Idle-Freq, CPU-Freq)

nmi_watchdog=0 modprobe.blacklist=pcspkr,iTCO_wdt,acpi_cpufreq,pcc_cpufreq,intel_cstate,intel_uncore,intel_powerclamp,i7core_edac,i5500_temp,coretemp,asus_atk0110 idle=halt intel_pstate=disable cpufreq.off=0

• None of the modules listed in the blacklist= should be listed by lsmod afterwards.

• To make CoreFreq the governor, you will ask for registration:

  insmod corefreqk.ko RDPMC_Enable=1 Register_ClockSource=1 Register_Governor=1 Register_CPU_Freq=1 Register_CPU_Idle=1

  From the UI, menu Settings, you can also do the registration.

• In short:

  modprobe.blacklist=iTCO_wdt

• Warmly recommended to add a new entry in your boot loader config as the BLACKLIST menu choice. Keep the original entry as a safe startup.

[chenkaigithub]

[cyring]

Thanks. I can see that there is no Vcore reading.

[cyring]

Vcore for SKL/X is stored at this line: https://github.com/cyring/CoreFreq/blob/220a0bd43508dbed60a51ebc527d7b6eb8db1873/corefreqk.c#L13609

... from register MSR_IA32_PERF_STATUS at: https://github.com/cyring/CoreFreq/blob/220a0bd43508dbed60a51ebc527d7b6eb8db1873/corefreqk.c#L13593

So far I have no clue which SKL/X register to read voltage VID from ?

[chenkaigithub]

I am not family with the CPU part. You can try other solutions, and I will help you to test them.

[cyring]

Also notice in screenshot that relative frequency and its ratio on stressed CPU 28 are around zero ! First time I'm encountering this issue. Can you show the Core view which displays the UCC URC TSC counters. Also can you show the Sensors and the Custom views.

I received once a CoreFreq execution report from Skylake/X and it was giving much more meaningful values than that.

EDIT: This i9-9980XE has also a CPUID 06_55 Stepping 4; but yours is a Xeon.

EDIT: Looking at the top-left LCD, we can see the frequency of 3678 MHz. But no such frequency in the first column of the lower area (and ratio stuck to 0.00) I believe you have to download a fresh copy of the source code (develop branch) and fully rebuild and reload all before making the screenshots. UI directives exist to remove some drawing areas, especially when facing a high core count processor. make help to list them. As an example of removing the bars drawing:

make NO_UPPER=1 clean all

You should then be able to watch your whole 48 CPUs, your Terminal height may however still need to be adjusted.

[chenkaigithub]

Is this OK now?

[cyring]

Is this OK now?

Yes much better. So no issue on this parr. Thank you.

Core voltage is the enhancement needed for your processor family.

[cyring]

Intel Xeon Skylake Processor Scalable Family Datasheets

• Volume 1

• Volume 2

[cyring]

No clue for Vcore.

289 for testings.

[cyring]

Closing the issue. Feel free to come back about the new version.