search

spdk / spdk

Storage Performance Development Kit
https://spdk.io/
Other
2.94k stars 1.16k forks source link

Is there any consideration for making spdk_mempool_create() populate the data buffers from memory other than the mempool structures #1994

Open nufosmatic opened 3 years ago

nufosmatic commented 3 years ago

Is there any consideration for making spdk_mempool_create() populate the data buffers from memory other than the mempool structures? Currently the socket_id is used to allocate memory for buffers and for data structures in the mem_pool.

Since the data structures are not touched by hardware, the data structures should be allocated from someplace other than the place the data buffers are allocated.

rte_mempool_create*() includes an "empty" and a "populate" that allow these memory structures to be allocated from other places.

Trying to operation across a NTB, and the data structures for the NTB-translated buffers end up on the other side. Very unhappy.

Suggest spdk_mempool_create_populate() with an additional parameter "socket_id_populate" from which the data buffers are populated.

Unfortunately, DPDK does not provide an exposure this way either, so some intermediate rte_mempool_create_populate() thingie has to be created.

SPDK 21.4 RH7.4 x86_64

jimharris commented 3 years ago

[Bug Scrub] This is something that could be added to the SPDK env layer. We need to provide APIs to create an empty mempool and then functions to populate it with application-allocated memory. User then just needs to call spdk_mem_register on the memory used to populate the mempool.