[Ubuntu 20.04] "ipmctl show -memoryresources" returns "One or more DIMMs have invalid PCD data" on Ice Lake Systems

sscargal commented 3 years ago

Issue Description

After creating an AppDirect or Memory Mode goal and rebooting, ipmctl show -memoryresources may report the following:

# ipmctl show -memoryresources
One or more DIMMs have invalid PCD data. A platform reboot is recommended to restore valid PCD data, then try again.

Other commands such as ipmctl show -dimm work.

Cause

Ubuntu 20.04 package repository ships ipmctl version 02.00.00.3709. On systems with newer PMem firmware (eg: 02.02.00.1553), this older version of ipmctl is incompatible.

The cause can be identified using ipmctl version:

# ipmctl version
Intel(R) Optane(TM) DC Persistent Memory Command Line Interface Version 02.00.00.3709
DCPMM 0x0001 supports FIS 2.4
DCPMM 0x0101 supports FIS 2.4
DCPMM 0x0201 supports FIS 2.4
DCPMM 0x0301 supports FIS 2.4
DCPMM 0x0011 supports FIS 2.4
DCPMM 0x0111 supports FIS 2.4
DCPMM 0x0211 supports FIS 2.4
DCPMM 0x0311 supports FIS 2.4
DCPMM 0x1001 supports FIS 2.4
DCPMM 0x1101 supports FIS 2.4
DCPMM 0x1201 supports FIS 2.4
DCPMM 0x1301 supports FIS 2.4
DCPMM 0x1011 supports FIS 2.4
DCPMM 0x1111 supports FIS 2.4
DCPMM 0x1211 supports FIS 2.4
DCPMM 0x1311 supports FIS 2.4
This ipmctl software version predates the firmware interface specification version (FIS | FWAPIVersion: 2.3) for 16 DCPMM(s). It is recommended to update ipmctl.

Solution

Building a later version of ipmctl from source solved the issue. See Building and Installing IPMCTL from Source on Linux.

Comments

Ubuntu needs to update the version of ipmctl delivered within their package repository to prevent this issue from causing widespread usability issues
ipmctl commands that need it, such as show memoryresources, should add the FIS check to help users easily identify these issues. It's not obvious that a user needs to run ipmctl version to perform a FIS version check.

StevenPontsler commented 3 years ago

Ubuntu 20.04 appears to be a long term support version. I can ask the maintainer about updating the version included though this has been a battle before. Usually they only accept things like security issues and then they want the same version just patched.

The FIS version is close enough that it should not present a problem. I suspect other changes resulted in the problem going away. I'll ask someone on my team to look into it.

StevenPontsler commented 3 years ago

Getting an update into 20.04 does not sound promising.

spawnflagger commented 2 years ago

Although Ubuntu 22.04 should be out soon, perhaps should consider adding newer (and compatible w/FW 02.02.00.1553) version of ipmctl package to focal-backports repo. Process documented here.

Many users will be on 20.04 for a while, since it's an LTS release (supported until April 2025). Then users would have to enable backports and run apt install ipmctl/focal-backports (much faster than installing build-essential and other dependencies needed to build from source)

Alternatively, Intel could maintain their own repo, similar to what Nvidia and Docker do. Then users would just need to add the intel repo in order to use apt to install latest version of ipmctl (and other intel packages).

spawnflagger commented 2 years ago

1 other note: The build documentation linked above, for Ubuntu 19+ says to sudo apt install ruby-asciidoctor ruby-asciidoctor-pdf , however on Ubuntu 20 those packages don't include the asciidoctor binary, so the build fails to create all the manpages. The fix is to also sudo apt install asciidoctor.

(pmem.io is on GitHub, however docs.pmem.io is a GitBook site, and I couldn't find anywhere to submit issue/feedback from there. I thought the frowny-face at the bottom might pop-up a dialog box to submit feedback, but it did not)

StevenPontsler commented 2 years ago

We will look into it. I am not familiar with the focal-backports.

@sscargal - can you update pmem.io?

sscargal commented 2 years ago

Yes, I'll update the documentation. Thanks for the info @spawnflagger

intel / ipmctl