Impossible to register IB memory on XPMEM shared memory

[GoogleCodeExporter]

It is impossible to register with IB some memory that has been attached with 
XPMEM.
On that test case, with centos 6.2, ibv_reg_mr fails with errno=EFAULT

Original issue reported on code.google.com by fzs...@gmail.com on 14 Jun 2012 at 8:30

Attachments:

• xpmem-ib-test.tar.bz2

[GoogleCodeExporter]

Some progress on understanding why ibv_reg_mr() returns EFAULT when attempting 
to register memory which was acquired via xpmem_attach().

From the 10K foot view
 xpmem_attach() sets vma->vm_flags |= VM_DONTCOPY | VM_RESERVED | VM_IO | VM_DONTEXPAND | VM_PFNMAP;

 Later when ibv_reg_mr() is called the ib_core module calls get_user_pages() on the VM range which xpmem_attached.
Since xpmem_attach() set VM_IO & VM_PFNMAP, get_user_pages() the first time 
thru the loop (i=0) trips on
if (!vma || (vma->vm_flags & (VM_IO | VM_PFNMAP)) ||  !(vm_flags & 
vma->vm_flags))
    return i ? : -EFAULT;

Original comment by stan3.1...@gmail.com on 27 Jun 2012 at 3:36

[GoogleCodeExporter]

This is a modified version of xpmem that works with IB. There is also a patch 
for IB to work with that version of xpmem. When doing a memory registration, 
the IBV_ACCESS_XPMEM flag must be added to ibv_reg_mr().

The function xpmem_get_user_pages() needs some locking; however it works well 
enough as a proof of concept.

Original comment by fzs...@gmail.com on 11 Jul 2012 at 9:05

Attachments:

• xpmem-for-ib.tar.bz2