Using DCPMM as Main Memory without DRAM Cache

[jieliangang]

I am referring to Section 2.3.1 of "Basic Performance Measurements of the Intel Optane DC Persistent Memory Module".

MM-Optane-Uncached In this configuration, we use uncached Optane DC as the system memory, that is, without DRAM caching Optane DC. This configuration represents a Optane DC system configuration where raw, uncached Optane DC is used as the main memory device. We include this configuration since the DRAM cache in MM-Optane- Cached obscures the raw performance of the Optane DC media — we do not expect this to be a common system configuration. To build this configuration, we configured the Optane DC PMM into App Direct mode and let the Linux kernel consider Optane DC to be DRAM. The kernel considers Optane DC to be slower memory and DRAM to be faster memory, and puts them in two separate NUMA nodes. Although it would be interesting to measure the performance when the whole system running directly on the NVMM, we cannot boot the operating system without any DRAM. Therefore, to run the tests, we configure applications to bind their memory to a NUMA node with exclusively Optane DC memory.

Instead of configuring DCPMM as Memory Mode, in which the DRAM would act as a last level cache, I would like to test with DCPMM as the main memory without the DRAM cache.

The author of the paper

"configured the Optane DC PMM into App Direct mode and let the Linux kernel consider Optane DC to be DRAM".

May I know how can the Linux kernel be configured to consider persistent memory to be DRAM?

I tried out this guide (How To Extend Volatile System Memory (RAM) using Persistent Memory on Linux), but my Linux kernel version (4.14) is <v5.1 which is required for the guide. It seems like I couldn't enable the migrate device model daxctl support.

This was also raised in #120 but I wasn't able to find a solution from there.

The machine in the paper is of Linux kernel version 4.13.0. Thus, there should be a way to configure the kernel (4.14) to consider persistent memory as DRAM. How can I configure DCPMM as main memory without the DRAM cache? Thanks!

[sscargal]

@jieliangang I confirmed with the paper authors that they used some custom code, and the feature they describe by MM-Optane-Uncached is not available in the mainline kernel.

As you found, the DAX-KMEM feature allows us to provision PMem through the Kernel, but this requires Kernel v5.2 or later. There is a Memory Tiering feature currently in development that allows the Kernel to manage both DRAM and PMem allocations, but this won't be available for a while.

A user-space solution can be achieved using MemKind, but this requires modifying the app to use the API.

Memverge has a solution that can use both DRAM and PMem for volatile allocations, plus they can snapshot, clone, and restore the memory resident image. Their product targets unmodified applications but also allows ISVs to use their APIs for more control.

[StevenPontsler]

Thanks for responding @sscargal I had no idea how to do it. Certainly ipmctl does not have a function for it.

[sheepx86]

Hi @jieliangang, I am a co-author of the paper you mentioned. At the time we were doing the experiments, the KMEM feature was not available yet. What we did was simply replace E820_TYPE_PMEM with E820_TYPE_RAM in the boot code (the locations in UEFI and BIOS boot code are different). It was mostly a hack but worked for our purposes.

I would strongly recommend you to switch to a newer kernel and use the DAX-KMEM feature now.

[jieliangang]

Thanks @sscargal and @sheepx86 for the tips and assistance!! I will switch to a newer kernel and try out the DAX-KMEM feature.

[sscargal]

@kjfbjweerjfn

There is no other application being run.

The Kernel/OS is running and will need the DRAM for internal buffers and the Page Cache.

[StevenPontsler]

Thanks for responding @sscargal -- it looks like this is a question about an article you are hosting. Is there a better place for questions about that to be asked?

[sscargal]

There is a PMem forum (https://groups.google.com/forum/#!forum/pmem) and #pmem Slack Channel where questions can be asked.