search

Mellanox / libvma

Linux user space library for network socket acceleration based on RDMA compatible network adaptors
https://www.mellanox.com/products/software/accelerator-software/vma?mtag=vma
Other
582 stars 153 forks source link

libvma doesn't support the cascaded epoll usage model #219

Open jiangdong-sun opened 8 years ago

jiangdong-sun commented 8 years ago

Tested on VMA_VERSION: 8.0.4-0 Release built on May 31 2016 15:34:50 OFED Version: MLNX_OFED_LINUX-3.3-1.0.0.0

I guess all versions of libvma don't support cascaded epoll usage model.

BTW, libkqueue use epoll as cascade model for monitoring EPOLLOUT event, so libkqueue cannot work well with libvma.

It's easy to show the issue. But if you want the test code, I can provide then.

OphirMunk commented 8 years ago

@jiangdong-sun any test code that can demo your issue is appreciated. Please attach

jiangdong-sun commented 8 years ago

The test code is attached here. t_cascaded_epoll.c.gz

How to verify

$ unset LD_PRELOAD $ ./t_cascaded_epoll host_ip host_port

The EPOLLIN event will be received after the connection succeeded.

But, with libvma.

$ export LD_PRELOAD=/usr/lib64/libvma.so $ ./t_cascaded_epoll host_ip host_port

The EPOLLIN event will never be triggerred even the connection succeeded

jiangdong-sun commented 8 years ago

@OphirMunk any feedback? Thanks!

OphirMunk commented 8 years ago

@jiangdong-sun Thanks for sharing your code which enables us to recreate your case. VMA is focused on low latency for RX/TX fast path. Is your cascaded epoll needed for low latency or not? Can the cascaded epoll be used on the "slow path" (for a specific TCP port). If yes - I would try using libvma.conf file to indicate the specific TCP port that should be handled by the Operating System (OS) and not by VMA. I would definitely would like to hear if you gave it a try and if it worked for you. You can find details on libvma.conf file here: http://www.mellanox.com/related-docs/prod_acceleration_software/VMA_8_0_4_User_Manual_DOC-00393.pdf (section 4.1) In addition - I am forwarding your issue to a Mellanox field application engineer that may ask you more about the demand/justification for using cascaded epoll.

jiangdong-sun commented 8 years ago

@OphirMunk Thanks for your reply. Yes, cascaded epoll is needed for low latency in our environment. Actually, it's exist in libkqueue used by our application. So the slow path doesn't meet our requirement. Hope you team can provide this feature, and I'd like to answer the questions from your field application engineer. Thanks.