jcarreira / cirrus-kv

High-performance key-value store
Apache License 2.0
12 stars 0 forks source link

RDMA memory registration #169

Open jcarreira opened 7 years ago

jcarreira commented 7 years ago

Right now the RDMA backend supports two methods for dealing with memory pinning:

  1. A pointer to memory (RDMAMem) previously pinned is passed to the RDMAClient to avoid having to register memory
  2. If no RDMAMem pointer is passed, Cirrus registers the memory region where the data passed to Cirrus lives

Currently, mostly version 2 is being used which is very expensive.

A potentially more efficient alternative would be to

  1. register a large memory region (e.g., 500MB) when a client is constructed
  2. serialize all writes and reads to that region by allocating memory from there (using external allocator provided by boost)
  3. if we run out of memory from that region we can resort to version 2 above
jcarreira commented 6 years ago

The externall allocator from boost doesn't seem very efficient when compared against libc's.

It might still be faster to pre-pin+BoostAllocator than to pin memory every time.