The goal is not applied after reboot.

Problem description: I tried to config the memory from AppDirect mode by creating a goal. But the goal is not applied after reboot. The detailed commands are as follows:

Motherboard: MD72-HB3. Two DIMMs of the Intel® Optane™ Persistent Memory 200 Series. OS: Ubuntu 20.04.2 LTS. ipmctl version: 02.00.00.3885. ndctl version: 67+. kernel version: 5.4.0-91-generic

The CPU info is:

Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   46 bits physical, 57 bits virtual
CPU(s):                          64
On-line CPU(s) list:             0-63
Thread(s) per core:              2
Core(s) per socket:              16
Socket(s):                       2
NUMA node(s):                    2
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           106
Model name:                      Intel(R) Xeon(R) Silver 4314 CPU @ 2.40GHz
Stepping:                        6
CPU MHz:                         799.886
CPU max MHz:                     3400.0000
CPU min MHz:                     800.0000
BogoMIPS:                        4800.00
Virtualization:                  VT-x
L1d cache:                       1.5 MiB
L1i cache:                       1 MiB
L2 cache:                        40 MiB
L3 cache:                        48 MiB
NUMA node0 CPU(s):               0-15,32-47
NUMA node1 CPU(s):               16-31,48-63
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Enhanced IBRS, IBPB conditional, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 ds_cpl vmx 
                                 smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 invpcid_single ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ep
                                 t_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local wbnoinvd dtherm ida a
                                 rat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq rdpid md_clear pconfig flush_l1d arch_capabilities

The bios info:

# dmidecode 3.2
Getting SMBIOS data from sysfs.
SMBIOS 3.3.0 present.
# SMBIOS implementations newer than version 3.2.0 are not
# fully supported by this version of dmidecode.

Handle 0x0000, DMI type 0, 26 bytes
BIOS Information
        Vendor: GIGABYTE
        Version: F10
        Release Date: 08/17/2021
        Address: 0xF0000
        Runtime Size: 64 kB
        ROM Size: 32 MB
        Characteristics:
                PCI is supported
                BIOS is upgradeable
                BIOS shadowing is allowed
                Boot from CD is supported
                Selectable boot is supported
                BIOS ROM is socketed
                EDD is supported
                Japanese floppy for NEC 9800 1.2 MB is supported (int 13h)
                Japanese floppy for Toshiba 1.2 MB is supported (int 13h)
                5.25"/360 kB floppy services are supported (int 13h)
                5.25"/1.2 MB floppy services are supported (int 13h)
                3.5"/720 kB floppy services are supported (int 13h)
                3.5"/2.88 MB floppy services are supported (int 13h)
                Print screen service is supported (int 5h)
                Serial services are supported (int 14h)
                Printer services are supported (int 17h)
                CGA/mono video services are supported (int 10h)
                USB legacy is supported
                BIOS boot specification is supported
                Targeted content distribution is supported
                UEFI is supported
        BIOS Revision: 5.22

Handle 0x0060, DMI type 13, 22 bytes
BIOS Language Information
        Language Description Format: Long
        Installable Languages: 1
                en|US|iso8859-1
        Currently Installed Language: en|US|iso8859-1

The command information is: pmctl show -memoryresources

 MemoryType   | DDR         | PMemModule  | Total       
========================================================
 Volatile     | 192.000 GiB | 0.000 GiB   | 192.000 GiB
 AppDirect    | -           | 0.000 GiB   | 0.000 GiB
 Cache        | 0.000 GiB   | -           | 0.000 GiB
 Inaccessible | 0.000 GiB   | 126.742 GiB | 126.742 GiB
 Physical     | 192.000 GiB | 126.742 GiB | 318.742 GiB

ipmctl create -goal PersistentMemoryType=AppDirect

The following configuration will be applied:
 SocketID | DimmID | MemorySize | AppDirect1Size | AppDirect2Size 
==================================================================
 0x0000   | 0x0300 | 0.000 GiB  | 126.000 GiB    | 0.000 GiB
Do you want to continue? [y/n] y
Created following region configuration goal
 SocketID | DimmID | MemorySize | AppDirect1Size | AppDirect2Size 
==================================================================
 0x0000   | 0x0300 | 0.000 GiB  | 126.000 GiB    | 0.000 GiB
A reboot is required to process new memory allocation goals.

After reboot. ipmctl show -goal

There are no goal configs defined in the system.
Please use 'show -region' to display currently valid persistent memory regions.

ipmctl show -memoryresources

 MemoryType   | DDR         | PMemModule  | Total       
========================================================
 Volatile     | 192.000 GiB | 0.000 GiB   | 192.000 GiB
 AppDirect    | -           | 0.000 GiB   | 0.000 GiB
 Cache        | 0.000 GiB   | -           | 0.000 GiB
 Inaccessible | 0.000 GiB   | 126.742 GiB | 126.742 GiB
 Physical     | 192.000 GiB | 126.742 GiB | 318.742 GiB

Can you post the output of "ipmctl show -pcd"?

Can you post the output of "ipmctl show -pcd"?

 MemoryType   | DDR         | PMemModule  | Total       
========================================================
 Volatile     | 192.000 GiB | 0.000 GiB   | 192.000 GiB
 AppDirect    | -           | 0.000 GiB   | 0.000 GiB
 Cache        | 0.000 GiB   | -           | 0.000 GiB
 Inaccessible | 0.000 GiB   | 126.742 GiB | 126.742 GiB
 Physical     | 192.000 GiB | 126.742 GiB | 318.742 GiB
root@super:/home/super# ipmctl show -pcd
--DimmID:0x0300--
--Pcd:LSA--
   Label Storage Area : Current Index
   Signature          : NAMESPACE_INDEX
   Flags              : 0x0
   LabelSize          : 0x1
   Sequence           : 0x2
   MyOffset           : 0x100
   MySize             : 0x100
   OtherOffset        : 0x0
   LabelOffset        : 0x200
   NumOfLabel         : 0x1fe
   Major              : 0x1
   Minor              : 0x2
   Checksum           : 0x2793e7b2d8323c1b
   Hexdump: For 64 bytes
   000:  ffffffffffffffff ffffffffffffffff ........ ........
   016:  ffffffffffffffff ffffffffffffffff ........ ........
   032:  ffffffffffffffff ffffffffffffffff ........ ........
   048:  ffffffffffffffff ffffffffffffffff ........ ........
--Pcd:Config--
   Table                     : PCD Config Header
   Signature                 : DMHD
   Length                    : 0x3c
   Revision                  : 0x13
   Checksum                  : 0x56
   OemId                     : INTEL 
   OemTableId                : INTEL 
   OemRevision               : 0x2
   CreatorId                 : INTL
   CreatorRevision           : 0x20091013
   CurrentConfDataSize       : 0x38
   CurrentConfStartOffset    : 0x3c
   ConfInputDataSize         : 0x88
   ConfInputDataOffset       : 0x74
   ConfOutputDataSize        : 0x88
   ConfOutputDataOffset      : 0xfc

   ---Table: PCD Current Config---
      Signature                 : CCUR
      Length                    : 0x38
      Revision                  : 0x13
      Checksum                  : 0x6
      OemId                     : ALASKA
      OemTableId                : GBTUAC
      OemRevision               : 0x2
      CreatorId                 : INTL
      CreatorRevision           : 0x20091013
      ConfigStatus              : 0x1
      VolatileMemSizeIntoSpa    : 0x0
      PersistentMemSizeIntoSpa  : 0x0

   ---Table: Platform Config Data Conf Input table---
      Signature                 : CIN_
      Length                    : 0x88
      Revision                  : 0x13
      Checksum                  : 0x81
      OemId                     : INTEL 
      OemTableId                : INTEL 
      OemRevision               : 0x2
      CreatorId                 : INTL
      CreatorRevision           : 0x20091013
      SequenceNumber             : 0x1

      PcatTable   : PCD Partition Size Change
      Type                      : 0x4
      Length                    : 0x10
      PartitionSizeChangeStatus : 0x0
      PartitionSize             : 0x1faf7f0000

      PcatTable   : PCD Interleave Info
      Type                      : 0x5
      Length                    : 0x48
      InterleaveSetIndex        : 0x1
      NumOfDimmsInInterleaveSet : 0x1
      InterleaveMemoryType      : 0x2
      InterleaveSizeChannel     : 0x40
      InterleaveSizeImc         : 0x40
      InterleaveChangeStatus    : 0x0

      IdentificationInfoTable   : PCD Identification Info
      DimmUniqueIdentifer       : 8089-a2-2116-00004268
      DimmLocation              : 0x30000
      PartitionOffset           : 0x0
      PmPartitionSize           : 0x1f80000000

   ---Table: Platform Config Data Conf Output table---
      Signature                 : COUT
      Length                    : 0x88
      Revision                  : 0x13
      Checksum                  : 0x46
      OemId                     : ALASKA
      OemTableId                : GBTUAC
      OemRevision               : 0x2
      CreatorId                 : INTL
      CreatorRevision           : 0x20091013
      SequenceNumber             : 0x1
      ValidationStatus           : 0x2

      PcatTable   : PCD Partition Size Change
      Type                      : 0x4
      Length                    : 0x10
      PartitionSizeChangeStatus : 0x0
      PartitionSize             : 0x1faf7f0000

      PcatTable   : PCD Interleave Info
      Type                      : 0x5
      Length                    : 0x48
      InterleaveSetIndex        : 0x1
      NumOfDimmsInInterleaveSet : 0x1
      InterleaveMemoryType      : 0x2
      InterleaveSizeChannel     : 0x40
      InterleaveSizeImc         : 0x40
      InterleaveChangeStatus    : 0x0

      IdentificationInfoTable   : PCD Identification Info
      DimmUniqueIdentifer       : 8089-a2-2116-00004268
      DimmLocation              : 0x30000
      PartitionOffset           : 0x0
      PmPartitionSize           : 0x1f80000000

The ConfigStatus is 1, which means BIOS succeeded in applying the goal. However, it is not mapping the memory for some reason.

Since you have a commercial BIOS, I am not that familiar with it. If you can capture BIOS logs, that will help in triaging the issue. There might be a warning there saying something like "non-POR configuration", but the App Direct rules in your manual indicate 1 PMem module only is allowed. https://download.gigabyte.com/FileList/Manual/server_mb_manual__mu72hb2_3_e_1.0.pdf?v=56cfb2def1fc79a3acaa8166eebe0e33

Assuming you have 6 x 32GB DDR (192GB), you have a supported 6+1 configuration. ipmctl show -topology should give you the physical DIMM slot numbers for both DDR and PMem, or look at dmidecode -t memory 'Locator' value.

I agree with Nolan that the PCD looks sane, so this appears to be a BIOS/Memory Training problem. Looking in the BIOS and BMC for platform related errors is the next best course of action. I recommend creating a support ticket with Gigabyte or your HW support vendor to get assistance in reviewing the logs, etc.

The 'Inaccessible' PMem capacity is due to one of the following:

Inaccessible: Total system persistent memory capacity that is inaccessible due to any of:

Platform configuration prevents accessing this capacity. For example, MemoryCapacity is configured but MemoryMode is not enabled by platform FW (current Memory Mode is 1LM).
Capacity is inaccessible because it is not mapped into the System Physical Address space (SPA). This is usually due to platform firmware memory alignment requirements.
Persistent capacity that is reserved. This capacity is the persistent memory partition capacity (rounded down for alignment) less any App Direct capacity. Reserved capacity typically results from a Memory Allocation Goal request that specified the Reserved property. This capacity is not mapped to System Physical Address space (SPA).
Capacity that is unusable because it has not been configured.
PMem module configured capacity but SKU prevents usage. For example, AppDirectCapacity but PMem module SKU is MemoryMode only.

You can check the BIOS e820 tables that provide memory layout information, eg:

# dmesg | egrep e820
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x0000000000099fff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009a000-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000e0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x0000000050079fff] usable
[    0.000000] BIOS-e820: [mem 0x000000005007a000-0x0000000050421fff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000050422000-0x0000000068bfefff] usable
[    0.000000] BIOS-e820: [mem 0x0000000068bff000-0x000000006ebfefff] reserved
[    0.000000] BIOS-e820: [mem 0x000000006ebff000-0x000000006f9fefff] ACPI NVS
[    0.000000] BIOS-e820: [mem 0x000000006f9ff000-0x000000006fffefff] ACPI data
[    0.000000] BIOS-e820: [mem 0x000000006ffff000-0x000000006fffffff] usable
[    0.000000] BIOS-e820: [mem 0x0000000070000000-0x000000008fffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fd000000-0x00000000fe7fffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fec00000-0x00000000fec00fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fec80000-0x00000000fed00fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fed40000-0x00000000fed44fff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000ff000000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x000000303fffffff] usable
[    0.000000] BIOS-e820: [mem 0x0000003040000000-0x000000305fffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000003060000000-0x000000ed5fffffff] persistent (type 7)
[    0.000000] BIOS-e820: [mem 0x000000ed60000000-0x0000011cffffffff] usable
[    0.000000] BIOS-e820: [mem 0x0000011d00000000-0x000001d9ffffffff] persistent (type 7)
[    0.000000] BIOS-e820: [mem 0x000001da00000000-0x000001da5fffffff] reserved

My system has two fully populated sockets with DRAM and PMem, so we see two persistent (type 7) entries. You should see one if everything in the BIOS works as expected.

Hi there. I am a colleague of @maaaay and I am currently working with this issue. We feel very sorry for the late reply.

Reply to @nolanhergert:

The ConfigStatus is 1, which means BIOS succeeded in applying the goal.

When raising this issue, @maaaay plugged 2x Optane PM 200 series DIMMs, which we considered unsupported by the motherboard after reviewing the vendor's documentation. We reconfigured our server following the instructions in GIGABYTE's documentation, and now ipmctl show -pcd returns a ConfigStatus of 0x6 (other outputs have no changes).

$ ipmctl show -pcd | grep -C 5 ConfigStatus
      OemId                     : ALASKA
      OemTableId                : GBTUAC
      OemRevision               : 0x2
      CreatorId                 : INTL
      CreatorRevision           : 0x20091013
      ConfigStatus              : 0x6
      VolatileMemSizeIntoSpa    : 0x0
      PersistentMemSizeIntoSpa  : 0x0

   ---Table: Platform Config Data Conf Input table---
      Signature                 : CIN_

We also tried ipmctl start -diagnostic Config which points out the problem more bluntly:

--Test = Config
   State = Failed
   --SubTest = PMem module specs
      State = Ok
   --SubTest = Duplicate PMem module
      State = Ok
   --SubTest = System Capability
      State = Ok
   --SubTest = Namespace LSA
      State = Ok
   --SubTest = PCD
      State = Failed
      Message.1 = The platform configuration check detected that the BIOS could not apply the configuration goal on PMem module 0x0300 for an unknown reason. The detailed status is COUT table status: (2 [UNKNOWN]), Partition change table status: (0), Interleave change table 1 status: (2), Interleave change table 2 status: (N/A).

Reply to @sscargal:

... or look at dmidecode -t memory 'Locator' value.

$ dmidecode -t memory | grep -C 17 'DIMM_P0_G0'
Handle 0x003F, DMI type 17, 92 bytes
Memory Device
        Array Handle: 0x0038
        Error Information Handle: Not Provided
        Total Width: 72 bits
        Data Width: 64 bits
        Size: 129728 MB
        Form Factor: DIMM
        Set: None
        Locator: DIMM_P0_G0
        Bank Locator: P0_Node0_Channel6_Dimm0
        Type: <OUT OF SPEC>
        Type Detail: Synchronous Non-Volatile LRDIMM
        Speed: 3200 MT/s
        Manufacturer: Intel
        Serial Number: 2116-00004268
        Asset Tag: DIMM_P0_G0_AssetTag
        Part Number: NMB1XXD128GPS
        Rank: 1
        Configured Memory Speed: 2666 MT/s
        Minimum Voltage: 1.2 V
        Maximum Voltage: 1.2 V
        Configured Voltage: 1.2 V
        Memory Technology: Intel persistent memory
        Memory Operating Mode Capability: Volatile memory Byte-accessible persistent memory
        Firmware Version: 1516
        Module Manufacturer ID: Bank 1, Hex 0x89
        Module Product ID: 0x4552
        Memory Subsystem Controller Manufacturer ID: Bank 1, Hex 0x89
        Memory Subsystem Controller Product ID: 0x097B
        Non-Volatile Size: None
        Volatile Size: 126 GB
        Cache Size: None
        Logical Size: None

You can check the BIOS e820 tables that provide memory layout information.

We checked the output of dmesg | grep e820 and unfortunately there isn't any persistent (type 7).

Did you try creating a new goal yet? ConfigStatus of 0x6 means:

6 – New configuration input structures have errors, old configuration used. Refer to the config output structures for additional errors.

To me, this seems to be a BIOS bug or BIOS Config issue.

The motherboard support page lists a newer BIOS and Firmware bundle available that may or may not help this situation. It would be worth updating both to eliminate any known bugs, especially as the BIOS delivers a Memory Voltage fix.

Going back to an earlier point, the motherboard manual states - "If system detects an un-validated POR configuration, then system issues a BIOS warning.". It would be nice to know what that warning/error is. You'll need to watch the console as the host boots and try to grab a snapshot. Additionally, the event should be logged by the platform and accessible via the web management interface. Read the Gigabyte Server Management Console manual for more instructions to configure, login, and check the logs.

With just one PMem module installed, the platform only supports AppDirect (not Memory Mode). By default, PMem ships with no config which defaults to Memory Mode. I wonder if the BIOS option that determines the mode is incorrectly set. I don't have access to a Gigabyte system and the manual doesn't provide a BIOS map, so you'll have to hunt around for the option, but it's usually somewhere under 'Advanced -> Memory Configuration' and it'll be called something like "Volatile Mode" with three options - Auto, 1LM/AppDirect, and 2LM/Memory Mode. You want to set this to 'Auto' so the BIOS reads the config off the PMem modules. You could try forcing the platform to '1LM/AppDirect' if 'Auto' doesn't work. Each OEM names the options differently and sometimes relocates them to different areas of the BIOS menu.

While you're in the BIOS, you should find the option to configure the PMem in either Memory Mode or AppDirect. I would certainly try using the BIOS interface to configure the platform rather than ipmctl to see if this makes any difference. You want to configure 100% for AppDirect and 0% for Memory Mode.

Thanks for the reply! We have successfully fixed the issue after following @sscargal's comments by updating the BIOS and the Firmware. The lesson learned is that BIOS version F10 and Firmware version 13.01 won't work for Optane Pmem 200 series.

We appreciate it very much for your kind advice.

intel / ipmctl

The goal is not applied after reboot. #182