post_send and post_recv will result in PageFault the first time an address is called on

[libertyeagle]

Due to ODP, post_send and post_recv will result in PageFault the first time an address is called on. This page fault will result in a huge overhead. We can avoid the overhead of post_recv by posting dummy sends to warm up the recv_mrs on the other side. However, post_send may not be easily solved.

As post_send sends messages constructed by user application, user app will allocate new space for their replys and requests. We can reuse the buffers for re-sending the same requests from a buffer. However, this might not be easily done for user app's server side.

[libertyeagle]

Maybe we can use ibv_advise_mr during SharedRegion::new()

[libertyeagle]

Issues:

• ODP receiver side recv_mr page fault
• ODP sender side post_send page fault
• RNR timer during warmup
• Client cloned replied message lead to significiant overhead
• check_income_connection in backend and server should be on seperate threads
• check_completion_queue in client side should be on seperate thread

[libertyeagle]

Current throughput with (not-complete) temp fix: ~37Gbps

[crazyboycjr]

Closes the issue since the throughput issue has been solved many months ago.