Fail to map in pmem mode

[fadingsoul]

QUESTION:

Details

I tried to map an pmem file using void *addr = pmem_map_file("/mnt/pmem8", 10000000000, PMEM_FILE_CREATE, 0666, &map_len, &is_pmem); while /mnt/pmem8 is mounted to /dev/pmem8 (with -o dax option)which is a fsdax device with ext4 file system created on it. But I always get is_pmem=0 with valid addr and map_len, which means it's not pmem mode. Did I do it wrong? Please help.

[pbalcer]

What's the kernel version?

[fadingsoul]

kernel version is 4.9.93, customized CentOS 7

[fadingsoul]

Besides, I can use the pointer addr later in my code as a pmem address and all the performance characteristics match pmem, so I'm pretty sure the address range is mapped to pmem device. Just curious why I got is_pmem=0 from pmem_map_file

[pbalcer]

The kernel that you are using does not support optimized flush DAX mappings that "is_pmem" flag is used to indicate. This support was added in kernel 4.15. This means that mapping is DAX, but the application must use msync() to synchronize the pages with the backing storage.

Anyway, the way the library detects if a mapping is pmem by relaying on MAP_SYNC and MAP_SHARED_VALIDATE. See man 2 mmap for more details.

http://man7.org/linux/man-pages//man2/munmap.2.html

[fadingsoul]

Thank you very much! I think this is the cause of my issue. I have two further questions

1. "the application must use msync() to synchronize the pages with the backing storage." Is it still safe to use clwb + fence / ntstore + fence to guarantee persistence(similiar to pmem_memcpy_persist)? or do I have to call extra msync()? what does “a more efficient way of making data modifications persistent” from mmap man page mean?
2. In my situation, what's the recommended way (fast and safe) to make data persist on pmem after the mmap from pmem_map_file?

[pbalcer]

1. Not on your version of the kernel. If you have to use this kernel, I'm afraid you are stuck with calling msync(). I strongly recommend upgrading. The more efficient way that the man page mentions is clwb + fence ;) But notice that it say that this is available since kernel 4.15.
2. Again, I strongly recommend upgrading the kernel. If you cannot do that, the only way you can make data persistent is to call msync() on fsdax.

[fadingsoul]

Based on my understanding, clwb/fence instruction work the same way in 4.15 and older kernels, what does 4.15 kernel change that makes clwb/fence safe? （or why it's not safe in older kernels?）

[pbalcer]

It's not about the the instructions but rather about whether they are safe to use or not. On older kernels they do not guarantee that the metadata file system is correct, which means that they do not guarantee that the data is actually persistent once flushed from userspace. Only if you have newer kernel and mapping with MAP_SYNC, the file system knows it has to ensure that once a page is writeable, it has to have its metadata flushed out to persistence. This is explained in the man page that I linked earlier.

See this blog post for more info: https://lwn.net/Articles/731706/

[fadingsoul]

Thank you very much, Piotr. I think the issue can be closed.