search

mtcp-stack / mtcp

mTCP: A Highly Scalable User-level TCP Stack for Multicore Systems
Other
1.98k stars 436 forks source link

Small number of batched events from stack to application #215

Open zhilongzheng opened 5 years ago

zhilongzheng commented 5 years ago

Hi,

I am tuning the performance of mTCP aware applications. However, I cannot get a relatively large improvement against Linux kernel based implementation.

The mTCP paper shows that a design to improve performance is Batched Event Handling. BUT, I found the returned number of events from mtcp_epoll_wait(mctx, events, -1) is small, which is stablely distributed and varied from 1 to ~64 (a lot of 1). Even through I stress the workload generator (wrk was used and varied Concurrency from 1024 to 8192), it seems no obvious increase on returned value of events.

I recorded the number of returned events on mTCP's two example applications (epserver.c & lighttpd-mtcp). Both reported a similiar result.

AND, I checked the flushed epoll events in FlushEpollEvents() function, it also printfs a small events number.

Only one CPU core was used to observe the above outputs and the default mTCP config file with adjusted core_mask was used.

My question is: Does such a small batch size of events have the potential to degrade end-to-end performance SIGNIFICANTLY?

My testing was conducted on a server with two 12-core CPUs and and Intel X710 NIC as mTCP server and 6 other servers as workload generators.

eunyoung14 commented 5 years ago

The number of batched events can vary depending on the the driver at the I/O layer and the size of your workload. The numbers in the paper were measured with PSIO which supports event-driven I/O scheduling. mTCP threads wake up less frequently and batch more events at a time. On the other hand, mTCP with DPDK polls packets from NICs, resulting in more frequent event handling/smaller batch size. However we didn't see much performance degradation when we moved from PSIO to DPDK.

zhilongzheng commented 5 years ago

Thank you for your quick reply!

I'll test more types of workloads