Open mesunyyam opened 3 years ago
Hello,
I solved the one issue that system-ram is immediately mapped into online memory. I modified udev rules and reload rules by following commands.
$ vim /etc/udev/rules.d/filename.rules $ udevadm control --reload-rules $ udevadm trigger --type=devices --action=change
However, I sill have the remain issue that system-ram is always mapped into target_node 0 (can't make PMEM only node). I am planning to look into BIOS setup.
Hi @mesunyyam, In a VMware virtual machine setup, I am also having the issue "system-ram is always mapped into target_node 0 (can't make PMEM only node)". Did you get a solution for this?.
I'm using Ubuntu 18.04.6 LTS with kernel version 5.3.1-multiclock, which comes from SyLab and just to make sure that the problem isn't caused by the kernel version, I'm getting the same error with version 5.15.17 of the kernel.
$ ndctl -v
66+
A while back, I used this kernel to successfully configure persistent memory as a separate NUMA node on a DELL R750 server following the steps. Now I am going to emulate persistent memory on a PC (with a 32G memory bank) and configure the emulated persistent memory as a separate NUMA node as well. Following the steps I set:
GRUB_CMDLINE_LINUX="memmap=24G!8G"
I got the following message after booting:
$ egrep '(DAX|PMEM)' /boot/config-`uname -r`
CONFIG_X86_PMEM_LEGACY_DEVICE=y
CONFIG_X86_PMEM_LEGACY=y
CONFIG_VIRTIO_PMEM=m
CONFIG_BLK_DEV_PMEM=m
CONFIG_NVDIMM_DAX=y
CONFIG_DAX_DRIVER=y
CONFIG_DAX=y
CONFIG_DEV_DAX=m
CONFIG_DEV_DAX_PMEM=m
CONFIG_DEV_DAX_KMEM=m
CONFIG_DEV_DAX_PMEM_COMPAT=m
CONFIG_FS_DAX=y
CONFIG_FS_DAX_PMD=y
CONFIG_ARCH_HAS_PMEM_API=y
$ ndctl list -N
[
{
"dev":"namespace0.0",
"mode":"fsdax",
"map":"mem",
"size":25769803776,
"sector_size":512,
"blockdev":"pmem0"
}
]
$ numactl -H
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
node 0 size: 7698 MB
node 0 free: 1933 MB
node distances:
node 0
0: 10
Then I did the following:
$ sudo ndctl destroy-namespace namespace0.0 --force
destroyed 0 namespaces
$ sudo ndctl create-namespace --mode=devdax --map=mem
{
"dev":"namespace0.0",
"mode":"devdax",
"map":"mem",
"size":"24.00 GiB (25.77 GB)",
"uuid":"5d7ff602-47c3-426c-9e26-80a5bc183ee7",
"daxregion":{
"id":0,
"size":"24.00 GiB (25.77 GB)",
"align":2097152,
"devices":[
{
"chardev":"dax0.0",
"size":"24.00 GiB (25.77 GB)",
"target_node":0,
"mode":"devdax"
}
]
},
"align":2097152
}
$ daxctl list
[
{
"chardev":"dax0.0",
"size":25767706624,
"target_node":0,
"mode":"devdax"
}
]
daxctl always map system-ram into target_node 0
If I try to do the next step:
$ daxctl reconfigure-device dax0.0 --mode=system-ram
libdaxctl: daxctl_dev_disable: dax0.0: error: device model is dax-class
dax0.0: disable failed: Operation not supported
error reconfiguring devices: Operation not supported
reconfigured 0 devices
Since node 0 exists and is being used by the system, I'm guessing that the target_node conflict is the cause of this error.
After that I set up 3 namespaces in kernel 5.15.17 by commanding :GRUB_CMDLINE_LINUX="memmap=4G!4G memmap=4G!16G memmap=4G!28G"
$ ndctl list -N
[
{
"dev":"namespace1.0",
"mode":"fsdax",
"map":"mem",
"size":4294967296,
"sector_size":512,
"blockdev":"pmem1"
},
{
"dev":"namespace0.0",
"mode":"fsdax",
"map":"mem",
"size":4294967296,
"sector_size":512,
"blockdev":"pmem0"
},
{
"dev":"namespace2.0",
"mode":"fsdax",
"map":"mem",
"size":4294967296,
"sector_size":512,
"blockdev":"pmem2"
}
]
I deleted each of these three and re-created them in devdax mode, and they are all still target_node 0.
$ daxctl list
[
{
"chardev":"dax1.0",
"size":4292870144,
"target_node":0,
"mode":"devdax"
},
{
"chardev":"dax2.0",
"size":4292870144,
"target_node":0,
"mode":"devdax"
},
{
"chardev":"dax0.0",
"size":4292870144,
"target_node":0,
"mode":"devdax"
}
]
Hello,
I am trying to change devdax dax0.0 device to PMEM only NUMA node by commanding as below.
$ daxctl reconfigure-device -m system-ram --no-online dax0.0
However, this system-ram is always mapped into target node 0 and created system-ram is always immediately online even though I specifically says "--no-online" in option. Desirable results are that newly created system-ram is mapped to target node 2 (since my server has 2 node) and this should be offline.
I checked kernel configuration as follows.
Is there another step that I need to do to solve this issue? (e.g. BIOS setting or additional kernel configurations?)
Thanks, Miseon Han