search

iqiyi / dpvs

DPVS is a high performance Layer-4 load balancer based on DPDK.
Other
3k stars 723 forks source link

Stress tests have a lot of errors #72

Closed NicoSpec closed 6 years ago

NicoSpec commented 6 years ago

we used tsung to test dpvs appearing lots of errors

Name |Highest Rate | Total number

error_abort_max_conn_retries | 99.3 / sec | 2557 error_abort_max_send_retries | 3.6 / sec | 73 error_connect_eaddrinuse | 21.7 / sec | 779 error_connect_etimedout | 133 / sec | 20857 error_connection_closed | 17.2 / sec | 360 error_timeout | 39.5 / sec | 811

ywc689 commented 6 years ago

How much traffic used for the stress tests? Was there any error logs in the system? Possibly not enough local IPs configured. Try 'dpip addr show' to find if there exists sa_miss in stress test. If so, configure more local IPs and try again.

beacer commented 6 years ago

Any update @NicoSpec . Can you show us the test env and configure? for example the client tool, server, DPVS's configure file. and dpvs's log.

NicoSpec commented 6 years ago

our dpvs is DR mode. our test tool is tsung,concurrency is 50,000,test page is 10 bytes. DPVS hardware: NIC is Intel Gigabit Ethernet. CPU is Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz.

DPVS's configure is default configure file.

dpvs's log have some error's log,for example: IPVS: dp_vs_sched_persist: persist-schedule: no dest found. IPVS: dp_vs_sched_persist: persist-schedule: no dest found. IPVS: dp_vs_sched_persist: persist-schedule: no dest found. IPVS: dp_vs_sched_persist: persist-schedule: no dest found. IPVS: dp_vs_sched_persist: persist-schedule: no dest found. IPVS: dp_vs_sched_persist: persist-schedule: no dest found. IPVS: dp_vs_sched_persist: persist-schedule: no dest found. IPVS: dp_vs_sched_persist: persist-schedule: no dest found. IPVS: dp_vs_sched_persist: persist-schedule: no dest found. IPVS: dp_vs_sched_persist: persist-schedule: no dest found. IPVS: dp_vs_sched_persist: persist-schedule: no dest found. IPVS: dp_vs_sched_persist: persist-schedule: no dest found.

This is tsung's error reports. default This is tsung's RPS reports: default

beacer commented 6 years ago

Can you show the output of dpip addr show, ipvsadm -ln, ipvsadm -G, dpip -s link show , dpip -s link show cpu ? The log persist-schedule: no dest found. could be warning not error, the tool we're using are wrk, ab and webbench, while wrk is highly recommend.

ywc689 commented 6 years ago

1no dest found indicates your backend servers may be the bottleneck of the stress test. Add more backend servers, or use high performance servers instead.

Are you really need persistent connections? If not, turn off it may improve performance.

NicoSpec commented 6 years ago

Hi, @beacer @ywc689 Now,we use "Intel(R) 10 Gigabit PCI Express Network" to test dpvs and close nginx's persistent connection.dpvs's log haven't any warning and tsung's report almost no error. PPS also reached the official test results. Thank you very much for your support.