Closed wwyf closed 1 year ago
According to Xeon scalable volume 2 , the IMC could be on PCI BUS:2
One way is like function Jasper_Forest_IMC()
...
https://github.com/cyring/CoreFreq/blob/6fcba8da6d8a2de75a248c040b14f3cb75869cd3/corefreqk.c#L4513
... where any memory controller is scanned on all CPU for a MEMBAR present on PCI Device and PCI Function identifiers
But MEMBAR appears NDA! Timings are not specified in vol 2.
What do you suggest to experiment and program IMC codes on your hardware ?
Thanks for your quick reply! It sounds like this issue can not be easily fixed by finding some key parameters in the document.
emmmmm... I also have no idea right now.
Linux:
|- Release [3.10.0-957.el7.x86_64]
If you boot an updated kernel we may read more lspci
identifiers
I suggest my live ISO which is Linux 5.15
based. This image will automatically compile/load/start CoreFreq
You can switch then to any VT for commands for PCI dump.
CoreFreq source code is within /opt/CoreFreq
if you are ready to experiment a new IMC implementation ?
I can't boot this image since this server is in the locked server room.
But there is another server using kernel 5.10.78
with the same hardware.
Let me try that server.
$ ./corefreq-cli -B -k -M
[ 0] Dell Inc.
[ 1] 2.5.4
[ 2] 01/13/2020
[ 3] Dell Inc.
[ 4] PowerEdge R740
[ 5]
[ 6] 4---D--
[ 7] SKU=NotProvided;ModelName=PowerEdge R740
[ 8] PowerEdge
[ 9] Dell Inc.
[10] 0WXD1Y
[11] A00
[12] .---5---.---V---0---0---
Linux:
|- Release [5.10.78+]
|- Version [#2 SMP Sun Nov 7 22:48:58 CST 2021]
|- Machine [x86_64]
Memory:
|- Total RAM 196325896 KB
|- Shared RAM 13388 KB
|- Free RAM 65936176 KB
|- Buffer RAM 23376 KB
|- Total High 0 KB
|- Free High 0 KB
CPU-Freq driver [ Missing]
Governor [ Missing]
CPU-Idle driver [ intel_idle]
|- Idle Limit [ C6]
|- State POLL C1 C1E C6
|- CPUIDLE MWAIT 0 MWAIT 0 MWAIT 0
|- Power -1 0 0 0
|- Latency 0 2 10 133
|- Residency 0 2 20 600
GenuineIntel [ 0]
$ ./corefreq-cli -s
Processor [Intel(R) Xeon(R) Gold 6230N CPU @ 2.30GHz]
|- PPIN# [75e862aaaa16b77b]
|- Architecture [Cascade Lake/X]
|- Vendor ID [GenuineIntel]
|- Microcode [0x0500002c]
|- Signature [ 06_55]
|- Stepping [ 7]
|- Online CPU [ 40/ 40]
|- Base Clock [ 99.760]
|- Frequency (MHz) Ratio
Min 997.66 < 10 >
Max 2294.61 < 23 >
|- Factory [100.000]
2300 [ 23 ]
|- Performance
|- P-State
TGT 3491.80 < 35 >
|- Turbo Boost [ UNLOCK]
1C 3491.80 < 35 >
2C 3292.26 < 33 >
3C 3192.50 < 32 >
4C 3192.50 < 32 >
5C 3092.73 < 31 >
6C 2893.20 < 29 >
7C 2893.20 < 29 >
8C 2893.20 < 29 >
9C 199.53 < 2 >
10C 399.06 < 4 >
12C 1197.19 < 12 >
13C 1596.25 < 16 >
14C 1995.31 < 20 >
15C 2394.37 < 24 >
16C 2793.44 < 28 >
|- Uncore [ UNLOCK]
Min 1197.19 < 12 >
Max 2394.37 < 24 >
|- TDP Level [ 0:2 ]
|- Programmable [ UNLOCK]
|- Configuration [ LOCK]
|- Turbo Activation [ LOCK]
Nominal 2294.61 [ 23 ]
Level1 1197.19 [ 12 ]
Level2 1197.19 [ 12 ]
Turbo AUTO [ 0 ]
Instruction Set Extensions
|- 3DNow!/Ext [N/N] ADX [Y] AES [Y] AVX/AVX2 [Y/Y]
|- AVX512-F [Y] AVX512-DQ [Y] AVX512-IFMA [N] AVX512-PF [N]
|- AVX512-ER [N] AVX512-CD [Y] AVX512-BW [Y] AVX512-VL [Y]
|- AVX512-VBMI [N] AVX512-VBMI2 [N] AVX512-VNMI [Y] AVX512-ALG [N]
|- AVX512-VPOP [N] AVX512-VNNIW [N] AVX512-FMAPS [N] AVX512-VP2I [N]
|- AVX512-BF16 [N] BMI1/BMI2 [Y/Y] CLWB [Y] CLFLUSH/O [Y/Y]
|- CLAC-STAC [Y] CMOV [Y] CMPXCHG8B [Y] CMPXCHG16B [Y]
|- F16C [Y] FPU [Y] FXSR [Y] LAHF-SAHF [Y]
|- MMX/Ext [Y/N] MON/MWAITX [Y/N] MOVBE [Y] PCLMULQDQ [Y]
|- POPCNT [Y] RDRAND [Y] RDSEED [Y] RDTSCP [Y]
|- SEP [Y] SHA [N] SSE [Y] SSE2 [Y]
|- SSE3 [Y] SSSE3 [Y] SSE4.1/4A [Y/N] SSE4.2 [Y]
|- SERIALIZE [N] SYSCALL [Y] SGX [N] RDPID [N]
Features
|- 1 GB Pages Support 1GB-PAGES [Capable]
|- Advanced Configuration & Power Interface ACPI [Capable]
|- Advanced Programmable Interrupt Controller APIC [Capable]
|- APIC Timer Invariance ARAT [Capable]
|- Core Multi-Processing CMP Legacy [Missing]
|- L1 Data Cache Context ID CNXT-ID [Missing]
|- Direct Cache Access DCA [Capable]
|- Debugging Extension DE [Capable]
|- Debug Store & Precise Event Based Sampling DS, PEBS [Capable]
|- CPL Qualified Debug Store DS-CPL [Capable]
|- 64-Bit Debug Store DTES64 [Capable]
|- Fast-String Operation Fast-Strings [Capable]
|- Fused Multiply Add FMA | FMA4 [Capable]
|- Hardware Lock Elision HLE [Capable]
|- Instruction Based Sampling IBS [Missing]
|- Instruction INVPCID INVPCID [Capable]
|- Long Mode 64 bits IA64 | LM [Capable]
|- LightWeight Profiling LWP [Missing]
|- Machine-Check Architecture MCA [Capable]
|- Memory Protection Extensions MPX [Capable]
|- Model Specific Registers MSR [Capable]
|- Memory Type Range Registers MTRR [Capable]
|- OS-Enabled Ext. State Management OSXSAVE [Capable]
|- Physical Address Extension PAE [Capable]
|- Page Attribute Table PAT [Capable]
|- Pending Break Enable PBE [Capable]
|- Process Context Identifiers PCID [Capable]
|- Perfmon and Debug Capability PDCM [Capable]
|- Page Global Enable PGE [Capable]
|- Page Size Extension PSE [Capable]
|- 36-bit Page Size Extension PSE36 [Capable]
|- Processor Serial Number PSN [Missing]
|- Resource Director Technology/PQE RDT-A [Capable]
|- Resource Director Technology/PQM RDT-M [Capable]
|- Restricted Transactional Memory RTM [Capable]
|- Safer Mode Extensions SMX [Capable]
|- Self-Snoop SS [Capable]
|- Supervisor-Mode Access Prevention SMAP [Capable]
|- Supervisor-Mode Execution Prevention SMEP [Capable]
|- Time Stamp Counter TSC [Invariant]
|- Time Stamp Counter Deadline TSC-DEADLINE [Capable]
|- TSX Force Abort MSR Register TSX-ABORT [Missing]
|- TSX Suspend Load Address Tracking TSX-LDTRK [Missing]
|- User-Mode Instruction Prevention UMIP [Missing]
|- Virtual Mode Extension VME [Capable]
|- Virtual Machine Extensions VMX [Capable]
|- Extended xAPIC Support x2APIC [ x2APIC]
|- Execution Disable Bit Support XD-Bit [Capable]
|- XSAVE/XSTOR States XSAVE [Capable]
|- xTPR Update Control xTPR [Capable]
Mitigation mechanisms
|- Indirect Branch Restricted Speculation IBRS [Capable]
|- Indirect Branch Prediction Barrier IBPB [Capable]
|- Single Thread Indirect Branch Predictor STIBP [Capable]
|- Speculative Store Bypass Disable SSBD [Capable]
|- Writeback & invalidate the L1 data cache L1D-FLUSH [Capable]
|- Hypervisor - No flush L1D on VM entry L1DFL_VMENTRY_NO [Capable]
|- Architectural - Buffer Overwriting MD-CLEAR [Capable]
|- Architectural - Rogue Data Cache Load RDCL_NO [Capable]
|- Architectural - Enhanced IBRS IBRS_ALL [Capable]
|- Architectural - Return Stack Buffer Alternate RSBA [Capable]
|- Architectural - Speculative Store Bypass SSB_NO [Capable]
|- Architectural - Microarchitectural Data Sampling MDS_NO [Capable]
|- Architectural - TSX Asynchronous Abort TAA_NO [Capable]
|- Architectural - Page Size Change MCE PSCHANGE_MC_NO [Capable]
|- Architectural - STLB QoS STLB [Missing]
|- Architectural - Functional Safety Island FuSa [Missing]
|- Architectural - RSM in CPL0 only RSM [Missing]
|- Architectural - Split Locked Access Exception SPLA [Missing]
|- Architectural - Snoop Filter QoS Mask SNOOP_FILTER [Missing]
Technologies
|- Data Cache Unit
|- L1 Prefetcher L1 HW < ON>
|- L1 IP Prefetcher L1 HW IP < ON>
|- L2 Prefetcher L2 HW < ON>
|- L2 Line Prefetcher L2 HW CL < ON>
|- System Management Mode SMM-Dual [OFF]
|- Hyper-Threading HTT [ ON]
|- SpeedStep EIST < ON>
|- Dynamic Acceleration IDA [ ON]
|- Turbo Boost TURBO < ON>
|- Energy Efficiency Optimization EEO <OFF>
|- Race To Halt Optimization R2H <OFF>
|- Watchdog Timer TCO < ON>
|- Virtualization VMX [ ON]
|- I/O MMU VT-d [OFF]
|- Version [ N/A]
|- Hypervisor [OFF]
|- Vendor ID [ N/A]
Performance Monitoring
|- Version PM [ 4]
|- Counters: General Fixed
| 4 x 48 bits 3 x 48 bits
|- Enhanced Halt State C1E <OFF>
|- C1 Auto Demotion C1A <OFF>
|- C3 Auto Demotion C3A <OFF>
|- C1 UnDemotion C1U <OFF>
|- C3 UnDemotion C3U <OFF>
|- C6 Core Demotion CC6 <OFF>
|- C6 Module Demotion MC6 <OFF>
|- Legacy Frequency ID control FID [OFF]
|- Legacy Voltage ID control VID [OFF]
|- P-State Hardware Coordination Feedback MPERF/APERF [ ON]
|- Hardware-Controlled Performance States HWP [OFF]
|- Hardware Duty Cycling HDC [OFF]
|- Package C-States
|- Configuration Control CONFIG [ LOCK]
|- Lowest C-State LIMIT < C6>
|- I/O MWAIT Redirection IOMWAIT < Enable>
|- Max C-State Inclusion RANGE < C1>
|- Core C-States
|- C-States Base Address BAR [ 0x514 ]
|- MONITOR/MWAIT
|- State index: #0 #1 #2 #3 #4 #5 #6 #7
|- Sub C-State: 0 2 0 2 0 0 0 0
|- Core Cycles [Capable]
|- Instructions Retired [Capable]
|- Reference Cycles [Capable]
|- Last Level Cache References [Capable]
|- Last Level Cache Misses [Capable]
|- Branch Instructions Retired [Capable]
|- Branch Mispredicts Retired [Capable]
|- Top-down slots Counter [Capable]
Power, Current & Thermal
|- Clock Modulation ODCM <Disable>
|- DutyCycle [ 0.00%]
|- Power Management PWR MGMT [ LOCK]
|- Energy Policy Bias Hint < 6>
|- Energy Policy HWP EPP [ 0]
|- Temperature Offset:Junction TjMax [ 0: 89 C]
|- Digital Thermal Sensor DTS [Capable]
|- Power Limit Notification PLN [Capable]
|- Package Thermal Management PTM [Capable]
|- Thermal Monitor 1 TM1 [ Enable]
|- Thermal Monitor 2 TM2 [Capable]
|- Thermal Design Power TDP [ 125 W]
|- Minimum Power Min [ 65 W]
|- Maximum Power Max [ 307 W]
|- Thermal Design Power Package < Enable>
|- Power Limit (1 sec) PL1 < 125 W>
|- Power Limit (1 sec) PL2 < 150 W>
|- Thermal Design Power Core <Disable>
|- Power Limit PL1 [Missing]
|- Thermal Design Power Uncore <Disable>
|- Power Limit PL1 [Missing]
|- Thermal Design Power DRAM <Disable>
|- Power Limit PL1 [Missing]
|- Thermal Design Power Platform <Disable>
|- Power Limit PL1 [Missing]
|- Power Limit PL2 [Missing]
|- Electrical Design Current EDC [Missing]
|- Thermal Design Current TDC [Missing]
|- Core Thermal Point
|- DTS Threshold #1 Threshold [Missing]
|- DTS Threshold #2 Threshold [Missing]
|- Package Thermal Point
|- DTS Threshold #1 Threshold [Missing]
|- DTS Threshold #2 Threshold [Missing]
|- Units
|- Power watt [ 0.125000000]
|- Energy joule [ 0.000061035]
|- Window second [ 0.000976562]
EDIT:
lspci -nn | grep -e "Controller" -e "Memory"
is probably masking the bus 2
Memory BAR is more often remapped from PCI root
I should run the command lspci -nn
?
$ lspci -nn
00:00.0 Host bridge [0600]: Intel Corporation Sky Lake-E DMI3 Registers [8086:2020] (rev 07)
00:04.0 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 07)
00:04.1 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 07)
00:04.2 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 07)
00:04.3 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 07)
00:04.4 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 07)
00:04.5 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 07)
00:04.6 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 07)
00:04.7 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 07)
00:05.0 System peripheral [0880]: Intel Corporation Sky Lake-E MM/Vt-d Configuration Registers [8086:2024] (rev 07)
00:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS [8086:2025] (rev 07)
00:05.4 PIC [0800]: Intel Corporation Sky Lake-E IOAPIC [8086:2026] (rev 07)
00:08.0 System peripheral [0880]: Intel Corporation Sky Lake-E Ubox Registers [8086:2014] (rev 07)
00:08.1 Performance counters [1101]: Intel Corporation Sky Lake-E Ubox Registers [8086:2015] (rev 07)
00:08.2 System peripheral [0880]: Intel Corporation Sky Lake-E Ubox Registers [8086:2016] (rev 07)
00:11.0 Unassigned class [ff00]: Intel Corporation C620 Series Chipset Family MROM 0 [8086:a1ec] (rev 09)
00:11.5 SATA controller [0106]: Intel Corporation C620 Series Chipset Family SSATA Controller [AHCI mode] [8086:a1d2] (rev 09)
00:14.0 USB controller [0c03]: Intel Corporation C620 Series Chipset Family USB 3.0 xHCI Controller [8086:a1af] (rev 09)
00:14.2 Signal processing controller [1180]: Intel Corporation C620 Series Chipset Family Thermal Subsystem [8086:a1b1] (rev 09)
00:16.0 Communication controller [0780]: Intel Corporation C620 Series Chipset Family MEI Controller #1 [8086:a1ba] (rev 09)
00:16.1 Communication controller [0780]: Intel Corporation C620 Series Chipset Family MEI Controller #2 [8086:a1bb] (rev 09)
00:16.4 Communication controller [0780]: Intel Corporation C620 Series Chipset Family MEI Controller #3 [8086:a1be] (rev 09)
00:17.0 SATA controller [0106]: Intel Corporation C620 Series Chipset Family SATA Controller [AHCI mode] [8086:a182] (rev 09)
00:1c.0 PCI bridge [0604]: Intel Corporation C620 Series Chipset Family PCI Express Root Port #1 [8086:a190] (rev f9)
00:1c.4 PCI bridge [0604]: Intel Corporation C620 Series Chipset Family PCI Express Root Port #5 [8086:a194] (rev f9)
00:1f.0 ISA bridge [0601]: Intel Corporation C621 Series Chipset LPC/eSPI Controller [8086:a1c1] (rev 09)
00:1f.2 Memory controller [0580]: Intel Corporation C620 Series Chipset Family Power Management Controller [8086:a1a1] (rev 09)
00:1f.4 SMBus [0c05]: Intel Corporation C620 Series Chipset Family SMBus [8086:a1a3] (rev 09)
00:1f.5 Serial bus controller [0c80]: Intel Corporation C620 Series Chipset Family SPI Controller [8086:a1a4] (rev 09)
01:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe [14e4:165f]
01:00.1 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM5720 2-port Gigabit Ethernet PCIe [14e4:165f]
02:00.0 PCI bridge [0604]: PLDA PCI Express Bridge [1556:be00] (rev 02)
03:00.0 VGA compatible controller [0300]: Matrox Electronics Systems Ltd. Integrated Matrox G200eW3 Graphics Controller [102b:0536] (rev 04)
16:00.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port A [8086:2030] (rev 07)
16:02.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port C [8086:2032] (rev 07)
16:05.0 System peripheral [0880]: Intel Corporation Sky Lake-E VT-d [8086:2034] (rev 07)
16:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 07)
16:05.4 PIC [0800]: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers [8086:2036] (rev 07)
16:08.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 07)
16:08.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 07)
16:08.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 07)
16:08.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 07)
16:08.4 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 07)
16:08.5 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 07)
16:08.6 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 07)
16:08.7 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 07)
16:09.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 07)
16:09.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 07)
16:09.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 07)
16:09.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 07)
16:09.4 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 07)
16:09.5 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 07)
16:09.6 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 07)
16:09.7 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 07)
16:0a.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 07)
16:0a.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 07)
16:0a.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 07)
16:0a.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 07)
16:0a.4 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 07)
16:0a.5 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 07)
16:0a.6 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 07)
16:0a.7 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 07)
16:0b.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 07)
16:0b.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 07)
16:0b.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 07)
16:0b.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 07)
16:0e.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 07)
16:0e.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 07)
16:0e.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 07)
16:0e.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 07)
16:0e.4 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 07)
16:0e.5 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 07)
16:0e.6 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 07)
16:0e.7 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 07)
16:0f.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 07)
16:0f.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 07)
16:0f.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 07)
16:0f.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 07)
16:0f.4 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 07)
16:0f.5 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 07)
16:0f.6 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 07)
16:0f.7 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 07)
16:10.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 07)
16:10.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 07)
16:10.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 07)
16:10.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 07)
16:10.4 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 07)
16:10.5 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 07)
16:10.6 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 07)
16:10.7 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 07)
16:11.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 07)
16:11.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 07)
16:11.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 07)
16:11.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 07)
16:1d.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2054] (rev 07)
16:1d.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2055] (rev 07)
16:1d.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2056] (rev 07)
16:1d.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2057] (rev 07)
16:1e.0 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2080] (rev 07)
16:1e.1 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2081] (rev 07)
16:1e.2 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2082] (rev 07)
16:1e.3 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2083] (rev 07)
16:1e.4 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2084] (rev 07)
16:1e.5 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2085] (rev 07)
16:1e.6 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2086] (rev 07)
17:00.0 Non-Volatile memory controller [0108]: Intel Corporation PCIe Data Center SSD [8086:0953] (rev 01)
18:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller [14e4:16d6] (rev 01)
18:00.1 Ethernet controller [0200]: Broadcom Inc. and subsidiaries BCM57412 NetXtreme-E 10Gb RDMA Ethernet Controller [14e4:16d6] (rev 01)
64:00.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port A [8086:2030] (rev 07)
64:02.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port C [8086:2032] (rev 07)
64:05.0 System peripheral [0880]: Intel Corporation Sky Lake-E VT-d [8086:2034] (rev 07)
64:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 07)
64:05.4 PIC [0800]: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers [8086:2036] (rev 07)
64:08.0 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2066] (rev 07)
64:09.0 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2066] (rev 07)
64:0a.0 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2040] (rev 07)
64:0a.1 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2041] (rev 07)
64:0a.2 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2042] (rev 07)
64:0a.3 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2043] (rev 07)
64:0a.4 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2044] (rev 07)
64:0a.5 System peripheral [0880]: Intel Corporation Sky Lake-E LM Channel 1 [8086:2045] (rev 07)
64:0a.6 System peripheral [0880]: Intel Corporation Sky Lake-E LMS Channel 1 [8086:2046] (rev 07)
64:0a.7 System peripheral [0880]: Intel Corporation Sky Lake-E LMDP Channel 1 [8086:2047] (rev 07)
64:0b.0 System peripheral [0880]: Intel Corporation Sky Lake-E DECS Channel 2 [8086:2048] (rev 07)
64:0b.1 System peripheral [0880]: Intel Corporation Sky Lake-E LM Channel 2 [8086:2049] (rev 07)
64:0b.2 System peripheral [0880]: Intel Corporation Sky Lake-E LMS Channel 2 [8086:204a] (rev 07)
64:0b.3 System peripheral [0880]: Intel Corporation Sky Lake-E LMDP Channel 2 [8086:204b] (rev 07)
64:0c.0 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2040] (rev 07)
64:0c.1 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2041] (rev 07)
64:0c.2 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2042] (rev 07)
64:0c.3 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2043] (rev 07)
64:0c.4 System peripheral [0880]: Intel Corporation Sky Lake-E Integrated Memory Controller [8086:2044] (rev 07)
64:0c.5 System peripheral [0880]: Intel Corporation Sky Lake-E LM Channel 1 [8086:2045] (rev 07)
64:0c.6 System peripheral [0880]: Intel Corporation Sky Lake-E LMS Channel 1 [8086:2046] (rev 07)
64:0c.7 System peripheral [0880]: Intel Corporation Sky Lake-E LMDP Channel 1 [8086:2047] (rev 07)
64:0d.0 System peripheral [0880]: Intel Corporation Sky Lake-E DECS Channel 2 [8086:2048] (rev 07)
64:0d.1 System peripheral [0880]: Intel Corporation Sky Lake-E LM Channel 2 [8086:2049] (rev 07)
64:0d.2 System peripheral [0880]: Intel Corporation Sky Lake-E LMS Channel 2 [8086:204a] (rev 07)
64:0d.3 System peripheral [0880]: Intel Corporation Sky Lake-E LMDP Channel 2 [8086:204b] (rev 07)
65:00.0 Non-Volatile memory controller [0108]: Intel Corporation NVMe Datacenter SSD [Optane] [8086:2701]
66:00.0 RAID bus controller [0104]: Broadcom / LSI MegaRAID SAS-3 3008 [Fury] [1000:005f] (rev 02)
b2:00.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port A [8086:2030] (rev 07)
b2:05.0 System peripheral [0880]: Intel Corporation Sky Lake-E VT-d [8086:2034] (rev 07)
b2:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 07)
b2:05.4 PIC [0800]: Intel Corporation Sky Lake-E IOxAPIC Configuration Registers [8086:2036] (rev 07)
b2:0e.0 Performance counters [1101]: Intel Corporation Sky Lake-E KTI 0 [8086:2058] (rev 07)
b2:0e.1 System peripheral [0880]: Intel Corporation Sky Lake-E UPI Registers [8086:2059] (rev 07)
b2:0f.0 Performance counters [1101]: Intel Corporation Sky Lake-E KTI 0 [8086:2058] (rev 07)
b2:0f.1 System peripheral [0880]: Intel Corporation Sky Lake-E UPI Registers [8086:2059] (rev 07)
b2:10.0 Performance counters [1101]: Intel Corporation Sky Lake-E KTI 0 [8086:2058] (rev 07)
b2:10.1 System peripheral [0880]: Intel Corporation Sky Lake-E UPI Registers [8086:2059] (rev 07)
b2:12.0 Performance counters [1101]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204c] (rev 07)
b2:12.1 Performance counters [1101]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204d] (rev 07)
b2:12.2 System peripheral [0880]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204e] (rev 07)
b2:12.4 Performance counters [1101]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204c] (rev 07)
b2:12.5 Performance counters [1101]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204d] (rev 07)
b2:15.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 07)
b2:15.1 Performance counters [1101]: Intel Corporation Sky Lake-E DDRIO Registers [8086:2088] (rev 07)
b2:16.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 07)
b2:16.1 Performance counters [1101]: Intel Corporation Sky Lake-E DDRIO Registers [8086:2088] (rev 07)
b2:16.4 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 07)
b2:16.5 Performance counters [1101]: Intel Corporation Sky Lake-E DDRIO Registers [8086:2088] (rev 07)
b2:17.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 07)
b2:17.1 Performance counters [1101]: Intel Corporation Sky Lake-E DDRIO Registers [8086:2088] (rev 07)
b3:00.0 Infiniband controller [0207]: Mellanox Technologies MT27700 Family [ConnectX-4] [15b3:1013]
Quoting the Scalable/X datasheet volume 2:
For Device 10 and 12 Functions 0-5 for offsets >= 256, PCIe extended configuration space are not designed for direct usage by OS or device drivers, and may not be accessible directly by OS components such as device drivers. The PCI Capability Pointer Register (CAPPTR) is set to a value of 40h. BIOS/firmware and/or BMC can access these registers, combine the information obtained with system implementation specifics, and if required, make it available to the OS through firmware and/or BMC interfaces.
This may sound like a tuff part.
Refreshing the old and unsolved issue #146
My notes: Source codes of interest:
I10NM_GET_IMC_BAR
base address resolved in function i10nm_get_ddr_munits
@wwyf : if you read the kernel debug trace when loading this EDAC driver, you may read the MMIO, next the IMC BAR; strings like:
socket# MMIO base
mc# MMIO base
Got it, I will give it a try.
This module is already loaded, and I find the log you want.
$ lsmod | grep edac
skx_edac 24576 0
nfit 77824 2 skx_edac
$ dmesg | grep -C 2 edac
[ 25.828843] XFS (dm-2): Ending clean mount
[ 25.828899] xfs filesystem being mounted at /home supports timestamps until 2038 (0x7fffffff)
[ 26.014371] EDAC MC0: Giving out device to module skx_edac controller Skylake Socket#0 IMC#0: DEV 0000:64:0a.0 (INTERRUPT)
[ 26.014435] EDAC MC1: Giving out device to module skx_edac controller Skylake Socket#0 IMC#1: DEV 0000:64:0c.0 (INTERRUPT)
[ 26.131185] intel_rapl_common: Found RAPL domain package
[ 26.131193] intel_rapl_common: Found RAPL domain dram
This log may correspond to this code.
To obtain the BAR address, read the value (in PCI configuration space) at Bus 0; Device 0; Function 0; Offset 48H and mask with the value 0x0007FFFFF8000.
sounds like something familiar
Intel® Xeon® Processor Scalable Memory Family Uncore Performance Monitoring Reference Manual
IMC0 - Channel 0 B2:D10:F2 0x2042
IMC0 - Channel 1 B2:D10:F6 0x2046
IMC0 - Channel 2 B2:D11:F2 0x204A
IMC1 - Channel 0 B2:D12:F2 0x2042
IMC1 - Channel 1 B2:D12:F6 0x2046
IMC1 - Channel 2 B2:D13:F2 0x204A
IMC Timings still not found, but DRAM Power could be improved.
This register is not implemented yet in CoreFreq
Edit: can you dump its value
rdmsr -ax 0x61C
Can you pull the develop
branch, refresh the whole CLI output corefreq-cli -s -n -m -n -B -n -k
... and also provide the Sensors > Power
view, with some CPU load in parallel; I'm interested in the RAM
energy consumed (left-bottom screen)
Got it. I will have a try.
Got it. I will have a try.
Be aware Skylake/X
is crashing with master
branch, and I believe your Cascade Lake/X
is having the same registers set.
Please check #374 about that issue.
Edit: Issue solved, you can now pull and try latest version.
The problem is the same as #160, though my CPU is Intel Gold 6230N.
Problems:
corefreq-cli -M
gives no output.This is some output of various commands about environment, may be they are useful.
(
-u
make output too long, so I upload the output as file) corefreq-cli -B -k -u -M.txt