search

mtcp-stack / mtcp

mTCP: A Highly Scalable User-level TCP Stack for Multicore Systems
Other
1.98k stars 436 forks source link

About mTCP's performance #191

Closed wtao0221 closed 6 years ago

wtao0221 commented 6 years ago

Hi,

On one of the two different node, I set up epserver (which runs in a VM) in single-process mode. And on the other node, I run epwget to generate 10000000 flows.

The stats shows to be 0.9M pps per-core, which seems not consistent with the Figure. 7 in the NSDI paper (https://www.usenix.org/system/files/conference/nsdi14/nsdi14-paper-jeong.pdf).

Could you please give me some suggestions on testing mTCP's performance?

eunyoung14 commented 6 years ago

Could you please tell us about your setup in more detail? epserver/epwget arguments, configurations, or # cpus?

mTCP is designed to perform the best on a number of concurrent connections. In particular, we evaluated the numbers in the paper with 1K concurrent connections per CPU core. Try to use the example arguments and configurations shown in apps/example/README and apps/example/epserver.conf.

By the way, the performance shown in the paper is evaluated with two separate physical machines.

wtao0221 commented 6 years ago

Both in epserver/epwget conf, I set the number of memory channels (i.e., num_mem_ch) to 1, and keep other parameters as default (rcvbuf = sndbuf = 8192, max_concurrency = 10000, max_num_buffers = 10000, tcp_timeouts=30, tcp_timewait=0).

I run epserver with argument -N 2 (2 logical cores) and run epwget with 10000000 -N 2 -c 8000 (10M requests, 2 cores and 8000 concurrency).

And the stats of epserver shows that the pps for all CPUs is ~1.8M.

I am not sure how can we calculate the message per second as shown in the paper. Can I just log down the start and end time of the program and calculate it? And the pps stat is given by DPDK, right? So the stats output of epserver can not directly serve as the performance of mTCP?

ajamshed commented 6 years ago

@wtao0221,

I see a number of issues with your setup. Like @eunyoung14 mentioned, please make sure that you run both endpoints on physical (bare-metal) machines. Also make sure that hyperthreading is turned off (each thread-pair should affinitize to a unique physical core). The micro-benchmark tests shown in Fig 7 were done using a program which was not released (they were based/extended on the source code from a third party that chose not to disclose the tool).

Your method of computing the message per second statistic may not be accurate. But I suggest that you write your own little program that can calculate the message transaction bandwidth of mTCP.

wtao0221 commented 6 years ago

@ajamshed

Thanks for your explanation.