external memory registration

fengggli commented 5 years ago

Tracks how to register external memory rte.

Most inspiration is from here
I tried dpdk v 19.05

fengggli commented 5 years ago

fengggli commented 5 years ago

Test output (following the instructions)

(base) fengggli@ribbit5(:):~/WorkSpace/dpdk905/dpdk-19.05$sudo ./build/app/testpmd -- --mp-alloc xmem
EAL: Detected 48 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:06:00.0 on NUMA socket 0
EAL:   probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:06:00.1 on NUMA socket 0
EAL:   probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:06:00.2 on NUMA socket 0
EAL:   probe driver: 8086:1521 net_e1000_igb
EAL: PCI device 0000:06:00.3 on NUMA socket 0
EAL:   probe driver: 8086:1521 net_e1000_igb
testpmd: No probed ethernet devices
testpmd: create a new mbuf pool <mbuf_pool_socket_0>: n=523456, size=2176, socket=0
testpmd: Allocated 2172MB of external memory
testpmd: preferred mempool ops selected: ring_mp_mc
testpmd: create a new mbuf pool <mbuf_pool_socket_1>: n=523456, size=2176, socket=1
testpmd: Allocated 2172MB of external memory
testpmd: preferred mempool ops selected: ring_mp_mc
Done
No commandline core given, start packet forwarding
io packet forwarding - ports=0 - cores=0 - streams=0 - NUMA support enabled, MP allocation mode: xmem

  io packet forwarding packets/burst=32
  nb forwarding cores=1 - nb forwarding ports=0
Press enter to exit

Telling cores to stop...
Waiting for lcores to finish...

  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
  RX-packets: 0              RX-dropped: 0             RX-total: 0
  TX-packets: 0              TX-dropped: 0             TX-total: 0
  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Done.

Bye...

fengggli commented 5 years ago

I don't necessarily need dpdk at this stage, I could just run mcas/testcl.cc in the virtualmachine instead(https://spdk.io/doc/env_8h.html#a0874731c44ac31e4b14d91c6844a87d1)

fengggli commented 5 years ago

I could use get_user_pages?(https://lwn.net/Articles/753027/, http://nuncaalaprimera.com/2014/using-hugepage-backed-buffers-in-linux-kernel-driver)

how pinned memory is managed in infiniband:

ib_umem_odp_map_dma_pages

how kernel handles MAP_HUGETLB:

user application: https://elixir.bootlin.com/linux/v4.15.18/source/mm/gup.c#L1088
in the kernel it's handled here (https://elixir.bootlin.com/linux/v4.15.18/source/mm/mmap.c#L1526) the second mmap(attach) in mcas is not a ANONYMOUS map, which will return error directly
dax (https://elixir.bootlin.com/linux/v4.15.18/source/drivers/dax/device.c#L536)

hugetlb_get_unmapped_area(https://elixir.bootlin.com/linux/v4.15.18/source/arch/x86/mm/hugetlbpage.c#L144)
from ldd3 book, previous edition, refer to the copy in my zotero
understanding linux kernel also has corresponding contents covering why we don't need to manipulate page tables directly since 2.6, mentioned in P418

fengggli commented 5 years ago

It turns out hugepage mapping's have several resitructions
1. For anonymous mappings:
2. For file_based mappings: underlying file has to be backed by a file, with fileoperations as , see the [shm-attach] example (https://elixir.bootlin.com/linux/v4.15.18/source/ipc/shm.c#L1380)
the libhugetlbfs is not quite useful here, since it's just a wrapper to use hugepage-mmap and hugepage-shm system calls.

What I could do is add a iocontrol in mcas (couldn't use mmap here, since directly mmap to non-hugepagefile(file which use the exact same fileoperations) will result unsuccessful mmap (https://elixir.bootlin.com/linux/v4.15.18/source/mm/mmap.c#L1507))

fengggli commented 5 years ago

how spdk do io using pmdk memory. (https://software.intel.com/en-us/articles/accessing-remote-persistent-memory-with-block-semantics-using-spdk-and-pmdk)
unlinked_fd example; https://github.com/libhugetlbfs/libhugetlbfs/blob/master/tests/shared.c
- mmap cannot operate on a non-hugetlbfs file descriptor
how spdk does it with pmdk??? (https://software.intel.com/en-us/articles/accessing-remote-persistent-memory-with-block-semantics-using-spdk-and-pmdk), it seems like direct read/write in bdev instead of mmaped regions?

Figure out how spdk block abstraction interacts with libpmemblk (where is the iomem?) Steps:

https://github.com/fengggli/spdk/blob/f85f7cb38ea02c854584ec8cc18239a48c5ca44f/lib/bdev/pmem/bdev_pmem.c#L200
instead of use spdk_nvme_ns_cmd_write (in blocknvme), bdev_pmem used spdk_bdev_io, i saw it will call pmemblk_write(pbp, buf, blockno) (https://github.com/fengggli/spdk/blob/f85f7cb38ea02c854584ec8cc18239a48c5ca44f/lib/bdev/pmem/bdev_pmem.c#L78); to interact with pmdk(buf is just virtual address i think), then a memory copy is performed! (https://github.com/pmem/pmdk/blob/1e676ffffe4190c858dfc2e1eadfcea0cd1a3952/src/examples/libpmemobj/pmemblk/obj_pmemblk.c#L282)

fengggli commented 5 years ago

pmap the test-hugepage-shm:

py36) lifen@sievert(:):~/Workspace/vagrantvm/vagrant-ubuntu18-spdk1810/linux-hwe-4.15.0$pmap 25474
25474:   ./src/components/store/nvmestore/testing/./test-hugepage-shm
0000000000400000      4K r-x-- test-hugepage-shm
0000000000600000      4K r---- test-hugepage-shm
0000000000601000      4K rw--- test-hugepage-shm
00000000017fc000    132K rw---   [ anon ]
00007f1cbac00000 262144K rw-s- SYSV00000002 (deleted)

test-mcas-nvmestore pmap: (they are stored in /dev/hugepages)


...
00007f0b8ce00000   2048K rw-s- nvme_comanchemap_32768
00007f0b8d000000   2048K rw-s- nvme_comanchemap_32769
00007f0b8d200000   2048K rw-s- nvme_comanchemap_32770
00007f0b8d400000   2048K rw-s- nvme_comanchemap_32771
00007f0b8d600000   2048K rw-s- nvme_comanchemap_32772
00007f0b8d800000   2048K rw-s- nvme_comanchemap_32773

...



* In name is passed in src/lib/core/src/dpdk.cpp then passed to dpdk

fengggli / dpdk