librpmem: rpmem_create failed to register memory.

tanghaodong25 commented 6 years ago

Hi,

We're testing librpmem with the 'pmdk/src/examples/librpmem/basic', hit error when creating pmem pool.

Testing Environment

Hardware

Mellanox ConnectX-4 + simulated pmem

Software

Ubuntu 18.04
Kernel version 4.15.0-21-generic
OFED version MLNX_OFED_LINUX-4.3-3.0.2.1-ubuntu18.04-x86_64
libfabrics version v1.6.1
PMDK version 1.4

Error log

rpmemd.log

registering memory: Unknown error -95

system log

[82047.691004] ib_umem_get: failed to get user pages, nr_pages=512, flags=1 [82047.691057] mlx5_1:mr_umem_get:766:(pid 63518): umem get failed (-95)

We digged into the code, found the error related to line 329 (pmdk/src/tools/rpmemd/rpmemd_fip.c). Can't register memory with PMEM.

 329         ret = fi_mr_reg(fip->domain, fip->addr, fip->size,
 330                         FI_REMOTE_READ | FI_REMOTE_WRITE, 0, 0, 0,
 331                         &fip->mr, NULL);
 332         if (ret) {
 333                 RPMEMD_FI_ERR(ret, "registering memory");
 334                 return -1;

zhouyuan commented 6 years ago

@lplewa is it possible to do more debug work with this "Unknown error -95"?

janekmi commented 6 years ago

I have found two issues:

on Ubuntu + kernel 4.15.0-21 ibv_reg_mr returns -EOPNOTSUPP for memory mapped from file created on regular fs (not device DAX, no fs with -odax). Issue does not occur on kernel 4.15.0-22 so this is the quick WA.
PMoF out-of-band protocol versioning pmem/issues#894

tanghaodong25 commented 6 years ago

The issue was fixed, thank you all.

pmem / issues