Closed maaaay closed 2 years ago
Can you post the output of "ipmctl show -pcd"?
Can you post the output of "ipmctl show -pcd"?
MemoryType | DDR | PMemModule | Total
========================================================
Volatile | 192.000 GiB | 0.000 GiB | 192.000 GiB
AppDirect | - | 0.000 GiB | 0.000 GiB
Cache | 0.000 GiB | - | 0.000 GiB
Inaccessible | 0.000 GiB | 126.742 GiB | 126.742 GiB
Physical | 192.000 GiB | 126.742 GiB | 318.742 GiB
root@super:/home/super# ipmctl show -pcd
--DimmID:0x0300--
--Pcd:LSA--
Label Storage Area : Current Index
Signature : NAMESPACE_INDEX
Flags : 0x0
LabelSize : 0x1
Sequence : 0x2
MyOffset : 0x100
MySize : 0x100
OtherOffset : 0x0
LabelOffset : 0x200
NumOfLabel : 0x1fe
Major : 0x1
Minor : 0x2
Checksum : 0x2793e7b2d8323c1b
Hexdump: For 64 bytes
000: ffffffffffffffff ffffffffffffffff ........ ........
016: ffffffffffffffff ffffffffffffffff ........ ........
032: ffffffffffffffff ffffffffffffffff ........ ........
048: ffffffffffffffff ffffffffffffffff ........ ........
--Pcd:Config--
Table : PCD Config Header
Signature : DMHD
Length : 0x3c
Revision : 0x13
Checksum : 0x56
OemId : INTEL
OemTableId : INTEL
OemRevision : 0x2
CreatorId : INTL
CreatorRevision : 0x20091013
CurrentConfDataSize : 0x38
CurrentConfStartOffset : 0x3c
ConfInputDataSize : 0x88
ConfInputDataOffset : 0x74
ConfOutputDataSize : 0x88
ConfOutputDataOffset : 0xfc
---Table: PCD Current Config---
Signature : CCUR
Length : 0x38
Revision : 0x13
Checksum : 0x6
OemId : ALASKA
OemTableId : GBTUAC
OemRevision : 0x2
CreatorId : INTL
CreatorRevision : 0x20091013
ConfigStatus : 0x1
VolatileMemSizeIntoSpa : 0x0
PersistentMemSizeIntoSpa : 0x0
---Table: Platform Config Data Conf Input table---
Signature : CIN_
Length : 0x88
Revision : 0x13
Checksum : 0x81
OemId : INTEL
OemTableId : INTEL
OemRevision : 0x2
CreatorId : INTL
CreatorRevision : 0x20091013
SequenceNumber : 0x1
PcatTable : PCD Partition Size Change
Type : 0x4
Length : 0x10
PartitionSizeChangeStatus : 0x0
PartitionSize : 0x1faf7f0000
PcatTable : PCD Interleave Info
Type : 0x5
Length : 0x48
InterleaveSetIndex : 0x1
NumOfDimmsInInterleaveSet : 0x1
InterleaveMemoryType : 0x2
InterleaveSizeChannel : 0x40
InterleaveSizeImc : 0x40
InterleaveChangeStatus : 0x0
IdentificationInfoTable : PCD Identification Info
DimmUniqueIdentifer : 8089-a2-2116-00004268
DimmLocation : 0x30000
PartitionOffset : 0x0
PmPartitionSize : 0x1f80000000
---Table: Platform Config Data Conf Output table---
Signature : COUT
Length : 0x88
Revision : 0x13
Checksum : 0x46
OemId : ALASKA
OemTableId : GBTUAC
OemRevision : 0x2
CreatorId : INTL
CreatorRevision : 0x20091013
SequenceNumber : 0x1
ValidationStatus : 0x2
PcatTable : PCD Partition Size Change
Type : 0x4
Length : 0x10
PartitionSizeChangeStatus : 0x0
PartitionSize : 0x1faf7f0000
PcatTable : PCD Interleave Info
Type : 0x5
Length : 0x48
InterleaveSetIndex : 0x1
NumOfDimmsInInterleaveSet : 0x1
InterleaveMemoryType : 0x2
InterleaveSizeChannel : 0x40
InterleaveSizeImc : 0x40
InterleaveChangeStatus : 0x0
IdentificationInfoTable : PCD Identification Info
DimmUniqueIdentifer : 8089-a2-2116-00004268
DimmLocation : 0x30000
PartitionOffset : 0x0
PmPartitionSize : 0x1f80000000
The ConfigStatus is 1, which means BIOS succeeded in applying the goal. However, it is not mapping the memory for some reason.
Since you have a commercial BIOS, I am not that familiar with it. If you can capture BIOS logs, that will help in triaging the issue. There might be a warning there saying something like "non-POR configuration", but the App Direct rules in your manual indicate 1 PMem module only is allowed. https://download.gigabyte.com/FileList/Manual/server_mb_manual__mu72hb2_3_e_1.0.pdf?v=56cfb2def1fc79a3acaa8166eebe0e33
Assuming you have 6 x 32GB DDR (192GB), you have a supported 6+1 configuration. ipmctl show -topology
should give you the physical DIMM slot numbers for both DDR and PMem, or look at dmidecode -t memory
'Locator' value.
I agree with Nolan that the PCD looks sane, so this appears to be a BIOS/Memory Training problem. Looking in the BIOS and BMC for platform related errors is the next best course of action. I recommend creating a support ticket with Gigabyte or your HW support vendor to get assistance in reviewing the logs, etc.
The 'Inaccessible' PMem capacity is due to one of the following:
Inaccessible: Total system persistent memory capacity that is inaccessible due to any of:
You can check the BIOS e820 tables that provide memory layout information, eg:
# dmesg | egrep e820
[ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x0000000000099fff] usable
[ 0.000000] BIOS-e820: [mem 0x000000000009a000-0x000000000009ffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x0000000050079fff] usable
[ 0.000000] BIOS-e820: [mem 0x000000005007a000-0x0000000050421fff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000050422000-0x0000000068bfefff] usable
[ 0.000000] BIOS-e820: [mem 0x0000000068bff000-0x000000006ebfefff] reserved
[ 0.000000] BIOS-e820: [mem 0x000000006ebff000-0x000000006f9fefff] ACPI NVS
[ 0.000000] BIOS-e820: [mem 0x000000006f9ff000-0x000000006fffefff] ACPI data
[ 0.000000] BIOS-e820: [mem 0x000000006ffff000-0x000000006fffffff] usable
[ 0.000000] BIOS-e820: [mem 0x0000000070000000-0x000000008fffffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000fd000000-0x00000000fe7fffff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000fec80000-0x00000000fed00fff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000fed40000-0x00000000fed44fff] reserved
[ 0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000303fffffff] usable
[ 0.000000] BIOS-e820: [mem 0x0000003040000000-0x000000305fffffff] reserved
[ 0.000000] BIOS-e820: [mem 0x0000003060000000-0x000000ed5fffffff] persistent (type 7)
[ 0.000000] BIOS-e820: [mem 0x000000ed60000000-0x0000011cffffffff] usable
[ 0.000000] BIOS-e820: [mem 0x0000011d00000000-0x000001d9ffffffff] persistent (type 7)
[ 0.000000] BIOS-e820: [mem 0x000001da00000000-0x000001da5fffffff] reserved
My system has two fully populated sockets with DRAM and PMem, so we see two persistent (type 7)
entries. You should see one if everything in the BIOS works as expected.
Hi there. I am a colleague of @maaaay and I am currently working with this issue. We feel very sorry for the late reply.
Reply to @nolanhergert:
The ConfigStatus is 1, which means BIOS succeeded in applying the goal.
When raising this issue, @maaaay plugged 2x Optane PM 200 series DIMMs, which we considered unsupported by the motherboard after reviewing the vendor's documentation.
We reconfigured our server following the instructions in GIGABYTE's documentation,
and now ipmctl show -pcd
returns a ConfigStatus of 0x6
(other outputs have no changes).
$ ipmctl show -pcd | grep -C 5 ConfigStatus
OemId : ALASKA
OemTableId : GBTUAC
OemRevision : 0x2
CreatorId : INTL
CreatorRevision : 0x20091013
ConfigStatus : 0x6
VolatileMemSizeIntoSpa : 0x0
PersistentMemSizeIntoSpa : 0x0
---Table: Platform Config Data Conf Input table---
Signature : CIN_
We also tried ipmctl start -diagnostic Config
which points out the problem more bluntly:
--Test = Config
State = Failed
--SubTest = PMem module specs
State = Ok
--SubTest = Duplicate PMem module
State = Ok
--SubTest = System Capability
State = Ok
--SubTest = Namespace LSA
State = Ok
--SubTest = PCD
State = Failed
Message.1 = The platform configuration check detected that the BIOS could not apply the configuration goal on PMem module 0x0300 for an unknown reason. The detailed status is COUT table status: (2 [UNKNOWN]), Partition change table status: (0), Interleave change table 1 status: (2), Interleave change table 2 status: (N/A).
Reply to @sscargal:
... or look at dmidecode -t memory 'Locator' value.
$ dmidecode -t memory | grep -C 17 'DIMM_P0_G0'
Handle 0x003F, DMI type 17, 92 bytes
Memory Device
Array Handle: 0x0038
Error Information Handle: Not Provided
Total Width: 72 bits
Data Width: 64 bits
Size: 129728 MB
Form Factor: DIMM
Set: None
Locator: DIMM_P0_G0
Bank Locator: P0_Node0_Channel6_Dimm0
Type: <OUT OF SPEC>
Type Detail: Synchronous Non-Volatile LRDIMM
Speed: 3200 MT/s
Manufacturer: Intel
Serial Number: 2116-00004268
Asset Tag: DIMM_P0_G0_AssetTag
Part Number: NMB1XXD128GPS
Rank: 1
Configured Memory Speed: 2666 MT/s
Minimum Voltage: 1.2 V
Maximum Voltage: 1.2 V
Configured Voltage: 1.2 V
Memory Technology: Intel persistent memory
Memory Operating Mode Capability: Volatile memory Byte-accessible persistent memory
Firmware Version: 1516
Module Manufacturer ID: Bank 1, Hex 0x89
Module Product ID: 0x4552
Memory Subsystem Controller Manufacturer ID: Bank 1, Hex 0x89
Memory Subsystem Controller Product ID: 0x097B
Non-Volatile Size: None
Volatile Size: 126 GB
Cache Size: None
Logical Size: None
You can check the BIOS e820 tables that provide memory layout information.
We checked the output of dmesg | grep e820
and unfortunately there isn't any persistent (type 7)
.
Did you try creating a new goal yet? ConfigStatus of 0x6 means:
6 – New configuration input structures have errors, old configuration used. Refer to the config output structures for additional errors.
To me, this seems to be a BIOS bug or BIOS Config issue.
The motherboard support page lists a newer BIOS and Firmware bundle available that may or may not help this situation. It would be worth updating both to eliminate any known bugs, especially as the BIOS delivers a Memory Voltage fix.
Going back to an earlier point, the motherboard manual states - "If system detects an un-validated POR configuration, then system issues a BIOS warning.". It would be nice to know what that warning/error is. You'll need to watch the console as the host boots and try to grab a snapshot. Additionally, the event should be logged by the platform and accessible via the web management interface. Read the Gigabyte Server Management Console manual for more instructions to configure, login, and check the logs.
With just one PMem module installed, the platform only supports AppDirect (not Memory Mode). By default, PMem ships with no config which defaults to Memory Mode. I wonder if the BIOS option that determines the mode is incorrectly set. I don't have access to a Gigabyte system and the manual doesn't provide a BIOS map, so you'll have to hunt around for the option, but it's usually somewhere under 'Advanced -> Memory Configuration' and it'll be called something like "Volatile Mode" with three options - Auto, 1LM/AppDirect, and 2LM/Memory Mode. You want to set this to 'Auto' so the BIOS reads the config off the PMem modules. You could try forcing the platform to '1LM/AppDirect' if 'Auto' doesn't work. Each OEM names the options differently and sometimes relocates them to different areas of the BIOS menu.
While you're in the BIOS, you should find the option to configure the PMem in either Memory Mode or AppDirect. I would certainly try using the BIOS interface to configure the platform rather than ipmctl to see if this makes any difference. You want to configure 100% for AppDirect and 0% for Memory Mode.
Thanks for the reply! We have successfully fixed the issue after following @sscargal's comments by updating the BIOS and the Firmware. The lesson learned is that BIOS version F10 and Firmware version 13.01 won't work for Optane Pmem 200 series.
We appreciate it very much for your kind advice.
Problem description: I tried to config the memory from AppDirect mode by creating a goal. But the goal is not applied after reboot. The detailed commands are as follows:
Motherboard: MD72-HB3. Two DIMMs of the Intel® Optane™ Persistent Memory 200 Series. OS: Ubuntu 20.04.2 LTS. ipmctl version: 02.00.00.3885. ndctl version: 67+. kernel version: 5.4.0-91-generic
The CPU info is:
The bios info:
The command information is: pmctl show -memoryresources
ipmctl create -goal PersistentMemoryType=AppDirect
After reboot. ipmctl show -goal
ipmctl show -memoryresources