Open fengggli opened 5 years ago
put is around 400IOPs, get is around 200IOPS (tried even with smaller io sizes), time is spent on check_exists, which uses map_keys call from metastore:
fixed in b133d2c661f61996a3a2ee97430e4d69fb3b2103
kvstore perf is consistent with the results in nvmestore-perf.xls (for 1M obj size, put 1.1 GB/s get-direct 2.6 GB/s)
1. server : O
2. client: OP_PUT
This failed:
https://github.com/fengggli/comanche/blob/9b37fd402c311ff919c8f57b3273014d0282ed50/src/servers/dawn/src/shard.cpp#L518
fi_mr_reg with the virtual address backed by a hugepage didn't success.
* how mapstore register memory to dawn server:
1. allocate memory(mapstore->lock)
https://github.com/fengggli/comanche/blob/9b37fd402c311ff919c8f57b3273014d0282ed50/src/components/store/mapstore/src/map_store.cpp#L280
2. register using the handler->ondemand_register above
* how dawn client register anonymous memory to fabric
https://github.com/fengggli/comanche/blob/9b37fd402c311ff919c8f57b3273014d0282ed50/src/components/client/dawn/unit_test/test1.cpp#L580-L586
* I tried dawn client with mapstore, passed the LargePutGetdirect(LargePutDirect) and LargeGetdirect(5096MiB/sec), (Using 11912 shard 0)
./src/components/client/dawn/unit_test/dawn-client-test1 --server-addr=10.0.0.82:11912
* I was trying to test the get_direct call from dawn client. I have tried:
1. use physical memory:alloc to allocate memory in mapstore, then I got same registration error in fi_reg_mr: this means spdk memory is not usable for fi_seg.
2. allocate using mmap(MAP_HUGETLB),and then register using spdk_mem_register. this failed, since its not reserved memory.
3. Try to use reserved memory-> test it with mapstore with dawnclient-> if it works, replace all iomemory allocatoin with register.
* Fixed in da0eeabd5d6bb0e7e5194ca609d41908df26cbd9
* error when start daw_test two times, even with mapstore.
Tracks the progress of dawn integration and exception handling
Notes for dawn.
Shards
Call stack src/main.cpp -> Dawn::Shard_launcher -> multiple Dawn::Shards.
memory