ipmctl can't list nvdimm in vm

[softblade-c]

Hi, I'm deploying DAOS in ESXi vm node. There's an Optane persistent memory installed in the host. nvdimms are added to the vm, but daos_server can't find any SCM at all, which invokes ipmctl to lisk the SCM. It seems that ipmctl doesn't work properly in a vm.

ndctl can list the nvdimms as follows:

[root@localhost ~]# ndctl list
[
  {
    "dev":"namespace2.0",
    "mode":"raw",
    "size":16777216,
    "sector_size":512,
    "blockdev":"pmem2"
  },
  {
    "dev":"namespace1.0",
    "mode":"raw",
    "size":6442450944,
    "sector_size":512,
    "blockdev":"pmem1"
  },
  {
    "dev":"namespace0.0",
    "mode":"raw",
    "size":6442450944,
    "sector_size":512,
    "blockdev":"pmem0"
  }
]

ipmctl finds nothing:

[root@localhost ~]# ipmctl show -dimm
Initialization failed. No PMem modules in the system.

The vm node has nvdimms allocated already. There's something different with the memory slot info:

[root@localhost ~]# dmidecode -t memory |grep -E "Locator:|Size"|grep -v Bank
        Maximum Memory Module Size: 32768 MB
        Maximum Total Memory Size: 491520 MB
        Installed Size: 4096 MB (Single-bank Connection)
        Enabled Size: 4096 MB (Single-bank Connection)
        Installed Size: 2048 MB (Single-bank Connection)
        Enabled Size: 2048 MB (Single-bank Connection)
        Installed Size: Not Installed
        Enabled Size: Not Installed
        ……
        Size: 4096 MB
        Locator: RAM slot #0
        Size: 2048 MB
        Locator: RAM slot #1
        Size: No Module Installed
        Locator: RAM slot #2
        Size: No Module Installed
        Locator: RAM slot #3
        ……
        Size: 6 GB
        Locator: NVD #0
        Size: 6 GB
        Locator: NVD #1
        Size: No Module Installed
        Locator: NVD #2
        ……
        Size: 16 MB
        Locator: NVD #18

The normal DRAM slot is RAM slot while the nvdimm resides in NVD. I'm not sure if this is the reason why ipmctl can't list nvdimm. What's the difference between ipmctl and ndctl? Can ipmctl work in vm?

Thanks.

[sscargal]

ipmctl cannot work within a VM as it requires access to the hardware, which in the case of ESXi is abstracted through virtualization. DAOS will need to find alternative methods for device discovery within a virtualized environment. ndctl could be one option - ndctl list -D to list 'NVDIMMs' for example.

[StevenPontsler]

Thanks Steve.

It looks like ndctl is listing namespaces which are an abstraction on top of PMems. The ipmctl utility works directly on the hardware modules so cannot work inside a VM.

[sscargal]

From the ESXi Hypervisor, the PMem should be exposed to the guest as vPMem devices to provide DirectAccess (DAX). See Using Persistent Memory in the ESXi documentation. There's more details in the VMWare docs for how to configure ESXi. Once the guest see's the virtual PMem devices, ndctl can be used within the guest.

[softblade-c]

Thanks Steve and Steven, Is it possible to refine ipmctl to work inside a VM just like ndctl does? After all, it's convenient to deploy and develop DAOS in VM environment. Or DAOS should be modified to use ndctl instead of ipmctl.

[sscargal]

That task is really for the hypervisors to implement the SMBUS and DDR-T protocols. Allowing communication over the SMBUS or DDR-T busses within the VM would be a security issue, so I don't see it happening.

[StevenPontsler]

Ipmctl expects to talk directly to the modules which means any changes it does would affect the whole system -- not just that particular VM. I believe this is why VM vendors have not implemented anything for this.

There maybe some useful information or functions that could be reused in another utility such as the code to read and interpret ACPI tables.

I would guess that it would be easier to use ndctl.