Memory Region Failure ibv_reg_mr failed [error code: 14]

agnesnatasya commented 2 years ago

Hi,

Problem

When setting up, I encountered ibv_reg_mr failed [error code: 14] error, indicating that the memory region requested is a bad / invalid address. Here are the values that I have

At libfs/lib/global/global.h
- g_log_size 32768UL // (128 MB)
At libfs/lib/rdma/mr.c
- mrs[0] = {addr = 140733004644352, length = 4299161600}
- mrs[1] = {addr = 140728713871360, length = 111071232}
At vi /etc/default/grub
- GRUB_CMDLINE_LINUX="memmap=8G!4G"
Size configuration
- python3 ./utils/change_dev_size.py 8 0 0
- At /etc/security/limits.conf
  - soft memlock unlimited
  - hard memlock unlimited

Do you mind helping me out with this problem?

Thank you!

wreda commented 2 years ago

This likely indicates that the memory you're trying to register is out-of-bounds.

Try setting your device size to a value lower than 8G using the change_dev_size.py script. After doing so, recompile LibFS/KernFS, and then rerun mkfs.sh.

agnesnatasya commented 2 years ago

You're correct, thank you very much for the suggestion and help!

Do you mind if I clarify why the problem occured? Assise's NVM region is 8GiB and from what I understand, the actual size of the emulated NVM is indeed 8GiB, as shown through ndctl list

{
  "dev":"namespace0.0",
  "mode":"fsdax",
  "map":"mem",
  "size":8589934592,
  "sector_size":512,
  "blockdev":"pmem0",
  "numa_node":0
}

wreda commented 2 years ago

I believe this is due to an alignment issue. I'll leave this open for now and will check later if there's a possible fix for this.

ut-osa / assise

Memory Region Failure ibv_reg_mr failed [error code: 14] #26

Problem