search

mtcp-stack / mtcp

mTCP: A Highly Scalable User-level TCP Stack for Multicore Systems
Other
1.98k stars 435 forks source link

mTCP throughput issue #260

Open lsjk opened 5 years ago

lsjk commented 5 years ago

Hello,

I am evaluating mTCP in virtual machine, and found the throughput is not comparable to kernel stack. Here's the VM configuration:

CPU: Intel(R) Xeon(R) Gold 61xx CPU Memory: 128G Number of core: 32 OS: centos7.2 Net: virtio-net

There are 2 VM, one is running mtcp, the other is running standard linux netstack

And I'm using iperf here https://github.com/thehajime/iperf-2.0.5-mtcp to do the testing, using single core, single connection.

for kernel sock (client) -> kernel socket (server) I got: 7.51 Gbits/sec for mtcp socket(client) -> kernel socket (server) I got: 5.04 Gbits/sec

That's about 70% of kernel stack performance. And the performance is extremely low when using the default sndbuf 8192. I changed it a much higher value. Because the iperf client always find the tcp send windows is full.

Is this expected behavior? Thank you very much.

ajamshed commented 5 years ago

@lsjk,

We haven't officially provided virtualization support for mTCP-dpdk. I will have to look into this issue in depth as I haven't tried this test before. However, just make sure you are using SRIOV. Please see this link for details: https://doc.dpdk.org/guides-18.11/nics/intel_vf.html

lsjk commented 4 years ago

@lsjk,

We haven't officially provided virtualization support for mTCP-dpdk. I will have to look into this issue in depth as I haven't tried this test before. However, just make sure you are using SRIOV. Please see this link for details: https://doc.dpdk.org/guides-18.11/nics/intel_vf.html

@ajamshed ,

Thank you very much for your reply. I will try it on physical setup. Another issue is, when I perf mtcp, I see the gettimeofday is on hot path and takes considerable time. I see gettimeofday is called on every mtcp main thread loop, is this only for arp timer ? If I hard-code dst mac address, is this still needed ? Thanks.

ajamshed commented 4 years ago

I believe gettimeofday is also being used to handle TCP timers. This is one of those system calls that we are heavily relying on for timing and synchronization issues.