search

iqiyi / dpvs

DPVS is a high performance Layer-4 load balancer based on DPDK.
Other
3.04k stars 730 forks source link

Unstable latency in DPVS #196

Closed haosdent closed 6 years ago

haosdent commented 6 years ago

Hi, when using DPVS. I found the latency is unstable compared to collect the real server directly.

My set up is 1 DPVS director and 1 DPVS real server with FULLNAT mode.

When I try to curl real server directly, the latency quite stable and keep less than 20 ms. But if I curl DPVS director server, the latency sometimes higher 200ms (about 1/5 requests) while almost requests latency keep under 20 ms. Do this result match the expectation? Is there any configuration could use to avoid this?

beacer commented 6 years ago

We do not found similar issue on our production env. It need more info, is there any ARP request/response on wan or lan side? is synproxy opened ? You can try to capture the packet on both client and server to see what's happened.

haosdent commented 6 years ago

Further check show it is because of ierrors cause TCP retransmission. Use testpmd show port xstats command could see most errors come from

rx_crc_errors: 7008

Would replace nic and cable to see if it resolve our problem.