search

erpc-io / eRPC

Efficient RPCs for datacenter networks
https://erpc.io/
Other
835 stars 137 forks source link

Improving performance using LTO and drivers #18

Closed vsbenas closed 5 years ago

vsbenas commented 5 years ago

I'm trying to improve performance of my eRPC application and would like to try using LTO instead of compiling eRPC as a library. Are there any instructions how to do so? Currently I'm using -lerpc when compiling my program.

Also, I'm getting a warning

WARNG: Modded driver unavailable. Performance will be low.

I assume using the provided drivers will increase the performance, but I can't find any guides how to do that as well. Could you help me out with this? I'm sure it will be helpful to other users as well.

NIC model: 56Gb Infiniband NIC (Mellanox MCX455A-FCAT PCIe-gen3 x16) Mellanox OFED or DPDK version: MLNX_OFED_LINUX-4.5-1.0.1.0-ubuntu18.04-x86_64 Operating system: Ubuntu 18.04

anujkaliaiitd commented 5 years ago

Currently, the examples in the apps/ folder use LTO. See: https://github.com/efficient/eRPC/blob/35270933836410d6779f9f2d4e1d8aaa1291a5d8/CMakeLists.txt#L109-L117 and https://github.com/efficient/eRPC/blob/35270933836410d6779f9f2d4e1d8aaa1291a5d8/CMakeLists.txt#L327-L329

If LTO is enabled, the eRPC library isn't compiled, and the app/ code is compiled along with the entire eRPC source. This isn't great, but I couldn't get LTO to work with a static library. Help is welcome :).

Modded drivers

We currently don't have modded drivers for libmlx5-based InfiniBand NICs like the above NIC model.

We have them for mlx4-based InfiniBand NICs (drivers/4.2/libmlx4-41mlnx1), or mlx5-based Ethernet NICs (drivers/4.4/libmlx5-41mlnx1). On Debian-based machines, these drivers can be installed using the update-driver.sh scripts.

Currently, I don't know how to sustainably manage the modded drivers, despite the simple optimizations. The problem is that Mellanox OFED isn't on GitHub, and Mellanox drivers from rdma_core (which is on GitHub), have fewer features