dawn integration and exception handling

[fengggli]

Tracks the progress of dawn integration and exception handling

Notes for dawn.

Shards

1. Call stack src/main.cpp -> Dawn::Shard_launcher -> multiple Dawn::Shards.

2. memory

[fengggli]

Performance:

• put is around 400IOPs, get is around 200IOPS (tried even with smaller io sizes), time is spent on check_exists, which uses map_keys call from metastore:

• fixed in b133d2c661f61996a3a2ee97430e4d69fb3b2103

• kvstore perf is consistent with the results in nvmestore-perf.xls (for 1M obj size, put 1.1 GB/s get-direct 2.6 GB/s)

[fengggli]

• kvstore-perf always failed with -elements=1, this is the problems of the benchmark tool itself.
• In kvstore-perf, if it is dawn memory_handle, the client will create io memory, and the memory_handle will be initialized, which causes _populate_pool_to_capacity to use put_direct,, which is not currently support by nvmestore

• Whats happens in put_direct?
  • in the client side, see, kvstore-perf
  • in the server side, see server/dawn. For now I could just do memory copy to nic memory(it can be easier, compared with register nic memory with nvmestore.)
  • it uses lock/unlock inside shards

[fengggli]

• get_direct error

  
  1. server : O
  2. client: OP_PUT

This failed: 
https://github.com/fengggli/comanche/blob/9b37fd402c311ff919c8f57b3273014d0282ed50/src/servers/dawn/src/shard.cpp#L518

fi_mr_reg with the virtual address backed by a hugepage didn't success.

* how mapstore register memory to dawn server:
  1. allocate memory(mapstore->lock)
  https://github.com/fengggli/comanche/blob/9b37fd402c311ff919c8f57b3273014d0282ed50/src/components/store/mapstore/src/map_store.cpp#L280
  2. register using the handler->ondemand_register above

* how dawn client register anonymous memory to fabric
https://github.com/fengggli/comanche/blob/9b37fd402c311ff919c8f57b3273014d0282ed50/src/components/client/dawn/unit_test/test1.cpp#L580-L586

* I tried dawn client with mapstore, passed the LargePutGetdirect(LargePutDirect) and LargeGetdirect(5096MiB/sec), (Using 11912 shard 0)

./src/components/client/dawn/unit_test/dawn-client-test1 --server-addr=10.0.0.82:11912

* I was trying to test the get_direct call from dawn client. I  have tried:
 1. use physical memory:alloc to allocate memory in mapstore, then I got same registration error in fi_reg_mr: this means spdk memory is not usable for fi_seg.
 2. allocate using mmap(MAP_HUGETLB),and then register using spdk_mem_register. this failed, since its not reserved memory.
 3. Try to use reserved memory-> test it with mapstore with dawnclient-> if it works, replace all iomemory allocatoin with register.

* Fixed in da0eeabd5d6bb0e7e5194ca609d41908df26cbd9
* error when start daw_test two times, even with mapstore.

[fengggli]

• Two stage get. (test failed for 1M object in dawn TWO_STAGE_THREATHHOLD= KiB(64)), in dawn unit test, mapstore passed
• get and get_direct has its own alignment, how to make value_length visible?