search

esnet / iperf

iperf3: A TCP, UDP, and SCTP network bandwidth measurement tool
Other
6.89k stars 1.27k forks source link

UDP iperf2 throughput cannot higher than 600Mbps #1217

Open ghost999991 opened 3 years ago

ghost999991 commented 3 years ago

Context

I'm troubleshooting the company product, but there's some issue with the UDP iperf2 throughput

Bug Report

I set my pc as the server and the product's kernal as the client, the bandwidth is 1100M:

For Iperf3 [ 5] local 192.168.1.2 port 5001 connected to 192.168.1.1 port 50031 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-1.00 sec 95.0 MBytes 797 Mbits/sec 0.053 ms 115/12278 (0.94%) [ 5] 1.00-2.00 sec 111 MBytes 931 Mbits/sec 0.066 ms 207/14407 (1.4%) [ 5] 2.00-3.00 sec 108 MBytes 908 Mbits/sec 0.073 ms 273/14130 (1.9%) [ 5] 3.00-4.00 sec 110 MBytes 923 Mbits/sec 0.067 ms 303/14384 (2.1%) [ 5] 4.00-5.00 sec 111 MBytes 934 Mbits/sec 0.056 ms 179/14425 (1.2%) [ 5] 5.00-6.00 sec 109 MBytes 913 Mbits/sec 0.054 ms 217/14140 (1.5%) [ 5] 6.00-7.00 sec 112 MBytes 943 Mbits/sec 0.053 ms 0/14394 (0%) [ 5] 7.00-8.00 sec 105 MBytes 883 Mbits/sec 0.057 ms 0/13466 (0%) [ 5] 8.00-9.00 sec 107 MBytes 900 Mbits/sec 0.056 ms 0/13734 (0%) iperf3: OUT OF ORDER - incoming packet = 68767 and received packet = 134302 AND SP = 5 [ 5] 9.00-10.00 sec 105 MBytes 879 Mbits/sec 0.075 ms 1/13416 (0.0075%) [ 5] 10.00-10.03 sec 3.45 MBytes 876 Mbits/sec 0.074 ms 0/441 (0%) [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-10.03 sec 0.00 Bytes 0.00 bits/sec 0.074 ms 1295/139215 (0.93%) [SUM] 0.0-10.0 sec 1 datagrams received out-of-order

-> It looks normal

For iperf2

[ 3] local 192.168.1.2 port 5001 connected with 192.168.1.1 port 44077 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 3] 0.0- 1.0 sec 66.4 MBytes 557 Mbits/sec 0.032 ms 22/47394 (0.046%) [ 3] 1.0- 2.0 sec 66.4 MBytes 557 Mbits/sec 0.027 ms 33/47425 (0.07%) [ 3] 2.0- 3.0 sec 66.5 MBytes 558 Mbits/sec 0.024 ms 24/47462 (0.051%) [ 3] 3.0- 4.0 sec 66.5 MBytes 558 Mbits/sec 0.029 ms 0/47460 (0%) [ 3] 4.0- 5.0 sec 66.5 MBytes 558 Mbits/sec 0.026 ms 0/47461 (0%) [ 3] 5.0- 6.0 sec 66.5 MBytes 558 Mbits/sec 0.030 ms 26/47463 (0.055%) [ 3] 6.0- 7.0 sec 66.5 MBytes 558 Mbits/sec 0.027 ms 35/47450 (0.074%) [ 3] 7.0- 8.0 sec 66.3 MBytes 556 Mbits/sec 0.028 ms 186/47452 (0.39%) [ 3] 8.0- 9.0 sec 66.5 MBytes 558 Mbits/sec 0.025 ms 0/47452 (0%) [ 3] 0.0-10.0 sec 665 MBytes 557 Mbits/sec 0.028 ms 359/474363 (0.076%)

-> with no packet loss, no additional commands; why the the throughput drop to 500Mbps?

Expected Behavior UDP can get 800~900Mbps throughput for iperf2 and iperf3, since I use cat5 cable for connection (1GB)

Actual Behavior Iperf2 get no more than 600Mbps, I have tried -l , -w , -P, but all of them not work.

Possible Solutions Unknown. Since I mainly use iperf2, I wish I can figure out this iperf2 issue.

davidBar-On commented 3 years ago

I am not familiar with iperf2 implementation), but it may be because of the burst mode that iperf3 supports. This feature allows sending UDP "bursts" of packets (I assume you use UDP) and the default is 10 packets per burst. I don't see similar feature in iperf2 (although I expected that -P would help).

To check this try running iperf3 with burst size of 1 packet by setting -b /1. If burst is the reason then the throughput should be comparable to iperf2. By the way, to test the maximum UDP throughput, try running iperf3 with -b /50.

ghost999991 commented 3 years ago

I am not familiar with iperf2 implementation), but it may be because of the burst mode that iperf3 supports. This feature allows sending UDP "bursts" of packets (I assume you use UDP) and the default is 10 packets per burst. I don't see similar feature in iperf2 (although I expected that -P would help).

To check this try running iperf3 with burst size of 1 packet by setting -b /1. If burst is the reason then the throughput should be comparable to iperf2. By the way, to test the maximum UDP throughput, try running iperf3 with -b /50.

Thank you for the suggestion! I have tried both /1 and /50, and their results are similar. When I tried reverse mode on client side (since I'm testing Down Link), it shows packet drop, but the server side which is listening shows no drop.

Burst mode DL testing

client# iperf3 -c 192.168.1.2 -i1 -p5001 -u -b1100M /1 -t5

Server listening on 5001 Accepted connection from 192.168.1.1, port 41784 [ 5] local 192.168.1.2 port 5001 connected to 192.168.1.1 port 44099 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-1.00 sec 109 MBytes 913 Mbits/sec 0.057 ms 79/14008 (0.56%) [ 5] 1.00-2.00 sec 109 MBytes 916 Mbits/sec 0.045 ms 45/14017 (0.32%) [ 5] 2.00-3.00 sec 109 MBytes 918 Mbits/sec 0.073 ms 15/14019 (0.11%) [ 5] 3.00-4.00 sec 109 MBytes 913 Mbits/sec 0.064 ms 88/14015 (0.63%) iperf3: OUT OF ORDER - incoming packet = 139 and received packet = 65674 AND SP = 5 (too many lines of this "OUT OF ORDER" info so I ignore some lines) iperf3: OUT OF ORDER - incoming packet = 569 and received packet = 66104 AND SP = 5 [ 5] 4.00-5.00 sec 109 MBytes 914 Mbits/sec 0.063 ms 90/14011 (0.64%) [ 5] 5.00-5.00 sec 0.00 Bytes 0.00 bits/sec 0.063 ms 0/0 (0%) [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-5.00 sec 0.00 Bytes 0.00 bits/sec 0.063 ms 317/70070 (0.45%) [SUM] 0.0- 5.0 sec 21 datagrams received out-of-order

client# iperf3 -c 192.168.1.2 -i1 -p5001 -u -b1100M /50 -t5

Server listening on 5001 Accepted connection from 192.168.1.1, port 41783 [ 5] local 192.168.1.2 port 5001 connected to 192.168.1.1 port 52075 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-1.00 sec 108 MBytes 906 Mbits/sec 0.032 ms 171/13999 (1.2%) [ 5] 1.00-2.00 sec 109 MBytes 914 Mbits/sec 0.064 ms 64/14014 (0.46%) [ 5] 2.00-3.00 sec 109 MBytes 916 Mbits/sec 0.049 ms 40/14020 (0.29%) [ 5] 3.00-4.00 sec 110 MBytes 920 Mbits/sec 0.055 ms 0/14032 (0%) iperf3: OUT OF ORDER - incoming packet = 143 and received packet = 65678 AND SP = 5 (too many lines of this "OUT OF ORDER" info, ignore some lines) iperf3: OUT OF ORDER - incoming packet = 663 and received packet = 66198 AND SP = 5 [ 5] 4.00-5.00 sec 108 MBytes 906 Mbits/sec 0.054 ms 228/14026 (1.6%) [ 5] 5.00-5.00 sec 0.00 Bytes 0.00 bits/sec 0.054 ms 0/0 (0%) [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-5.00 sec 0.00 Bytes 0.00 bits/sec 0.054 ms 503/70091 (0.72%) [SUM] 0.0- 5.0 sec 31 datagrams received out-of-order

--> Both results seem similar.

Reverse mode

client# iperf3 -c 192.168.1.2 -i1 -p5001 -u -b1100M /1 -t5 -R Connecting to host 192.168.1.2, port 5001 Reverse mode, remote host 192.168.1.2 is sending [ 4] local 192.168.1.1 port 46907 connected to 192.168.1.2 port 5001 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 0.00-1.00 sec 84.3 MBytes 707 Mbits/sec 0.009 ms 1939/12725 (15%) [ 4] 1.00-2.00 sec 90.7 MBytes 760 Mbits/sec 1.626 ms 2888/14496 (20%) [ 4] 2.00-3.00 sec 91.8 MBytes 769 Mbits/sec 0.018 ms 1953/13697 (14%) [ 4] 3.00-4.00 sec 91.0 MBytes 763 Mbits/sec 0.017 ms 2689/14333 (19%) [ 4] 4.00-5.00 sec 92.1 MBytes 772 Mbits/sec 0.020 ms 2827/14613 (19%) [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 0.00-5.00 sec 552 MBytes 925 Mbits/sec 0.020 ms 12296/69864 (18%) [ 4] Sent 69864 datagrams

I have also test the TCP results, it can only get 300Mbps, and its RTT is 20ms. I'm wondering is it possible the link delay cause the package drop?

davidBar-On commented 3 years ago

The burst rate doesn't have an effect probably because you added a blank before the / (for some reason iper3 doesn't detect that error). In this case the / is handled as another parameter and is just ignored along with the number that follows it.

Did you have the -b110M also in the original test? When bit rate is set, the default burst size is 1, so if this was the case the burst does not explain the difference between iperf2 and iperf3.

If any case I suggest that you will try running with setting the burst, but with bit rate set to 200M, as the burst may increase throughput over 110M. I therefore suggest that you will try with -b 200M/1, -b 200M/10, -b 200M/50.

ghost999991 commented 3 years ago

The burst rate doesn't have an effect probably because you added a blank before the / (for some reason iper3 doesn't detect that error). In this case the / is handled as another parameter and is just ignored along with the number that follows it.

Did you have the -b110M also in the original test? When bit rate is set, the default burst size is 1, so if this was the case the burst does not explain the difference between iperf2 and iperf3.

If any case I suggest that you will try running with setting the burst, but with bit rate set to 200M, as the burst may increase throughput over 110M. I therefore suggest that you will try with -b 200M/1, -b 200M/10, -b 200M/50.

Here's the result with no blank before /

client# iperf3 -c 192.168.1.2 -i1 -p5001 -u -b1100M/1 -t5 server# iperf3 -s -i1 -p5001 Accepted connection from 192.168.1.1, port 55404 [ 5] local 192.168.1.2 port 5001 connected to 192.168.1.1 port 40527 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-1.00 sec 91.7 MBytes 769 Mbits/sec 0.057 ms 32/11768 (0.27%) [ 5] 1.00-2.00 sec 104 MBytes 869 Mbits/sec 0.054 ms 273/13533 (2%) [ 5] 2.00-3.00 sec 109 MBytes 916 Mbits/sec 0.057 ms 458/14431 (3.2%) [ 5] 3.00-4.00 sec 113 MBytes 946 Mbits/sec 0.057 ms 0/14429 (0%) iperf3: OUT OF ORDER - incoming packet = 57 and received packet = 65592 AND SP = 5 iperf3: OUT OF ORDER - incoming packet = 63 and received packet = 65598 AND SP = 5 iperf3: OUT OF ORDER - incoming packet = 77 and received packet = 65612 AND SP = 5 [ 5] 4.00-5.00 sec 111 MBytes 934 Mbits/sec 0.054 ms 182/14433 (1.3%) [ 5] 5.00-5.02 sec 2.45 MBytes 927 Mbits/sec 0.034 ms 0/313 (0%) [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-5.02 sec 0.00 Bytes 0.00 bits/sec 0.034 ms 945/68907 (1.4%) [SUM] 0.0- 5.0 sec 3 datagrams received out-of-order

client# iperf3 -c 192.168.1.2 -i1 -p5001 -u -b1100M/50 -t5 server# iperf3 -s -i1 -p5001 Accepted connection from 192.168.1.1, port 55405 [ 5] local 192.168.1.2 port 5001 connected to 192.168.1.1 port 57157 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-1.00 sec 108 MBytes 908 Mbits/sec 0.115 ms 0/13851 (0%) [ 5] 1.00-2.00 sec 112 MBytes 939 Mbits/sec 0.074 ms 0/14325 (0%) [ 5] 2.00-3.00 sec 110 MBytes 922 Mbits/sec 0.120 ms 242/14313 (1.7%) [ 5] 3.00-4.00 sec 111 MBytes 935 Mbits/sec 0.057 ms 57/14325 (0.4%) [ 5] 4.00-5.00 sec 112 MBytes 939 Mbits/sec 0.105 ms 0/14325 (0%) [ 5] 5.00-5.04 sec 3.98 MBytes 930 Mbits/sec 0.107 ms 0/510 (0%) [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-5.04 sec 0.00 Bytes 0.00 bits/sec 0.107 ms 299/71649 (0.42%)

Here's the result for 110M/1, 110M/10

client# iperf3 -c 192.168.1.2 -i1 -p5001 -u -b 110M/1 -t5 server side Accepted connection from 192.168.1.1, port 55412 [ 5] local 192.168.1.2 port 5001 connected to 192.168.1.1 port 42484 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-1.00 sec 9.59 MBytes 80.2 Mbits/sec 0.263 ms 300/1527 (20%) [ 5] 1.00-2.01 sec 11.7 MBytes 97.6 Mbits/sec 0.284 ms 180/1678 (11%) [ 5] 2.01-3.00 sec 8.05 MBytes 67.8 Mbits/sec 0.065 ms 657/1687 (39%) [ 5] 3.00-4.00 sec 8.48 MBytes 71.3 Mbits/sec 0.263 ms 586/1671 (35%) [ 5] 4.00-5.01 sec 8.36 MBytes 69.5 Mbits/sec 0.047 ms 616/1686 (37%) [ 5] 5.01-5.04 sec 0.00 Bytes 0.00 bits/sec 0.047 ms 0/0 (0%) [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-5.04 sec 0.00 Bytes 0.00 bits/sec 0.047 ms 2339/8249 (28%)

client# iperf3 -c 192.168.1.2 -i1 -p5001 -u -b 110M/10 -t5 server side Accepted connection from 192.168.1.1, port 55413 [ 5] local 192.168.1.2 port 5001 connected to 192.168.1.1 port 51551 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-1.02 sec 5.41 MBytes 44.7 Mbits/sec 0.299 ms 839/1532 (55%) [ 5] 1.02-2.00 sec 12.1 MBytes 103 Mbits/sec 0.069 ms 134/1688 (7.9%) [ 5] 2.00-3.01 sec 10.9 MBytes 91.0 Mbits/sec 0.333 ms 273/1670 (16%) [ 5] 3.01-4.00 sec 6.36 MBytes 53.7 Mbits/sec 0.064 ms 866/1680 (52%) [ 5] 4.00-5.01 sec 8.34 MBytes 69.3 Mbits/sec 0.065 ms 613/1680 (36%) [ 5] 5.01-5.04 sec 0.00 Bytes 0.00 bits/sec 0.065 ms 0/0 (0%) [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-5.04 sec 0.00 Bytes 0.00 bits/sec 0.065 ms 2725/8250 (33%)

Here's the result for 200M/1, 200M/10, and 200M/20

client# iperf3 -c 192.168.1.2 -i1 -p5001 -u -b 200M/1 -t5 server side Accepted connection from 192.168.1.1, port 55406 [ 5] local 192.168.1.2 port 5001 connected to 192.168.1.1 port 53297 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-1.00 sec 16.3 MBytes 136 Mbits/sec 0.236 ms 742/2823 (26%) [ 5] 1.00-2.00 sec 17.4 MBytes 146 Mbits/sec 0.061 ms 824/3050 (27%) [ 5] 2.00-3.00 sec 20.1 MBytes 169 Mbits/sec 0.085 ms 480/3052 (16%) [ 5] 3.00-4.00 sec 16.6 MBytes 139 Mbits/sec 0.110 ms 931/3053 (30%) [ 5] 4.00-5.00 sec 17.9 MBytes 150 Mbits/sec 0.069 ms 763/3051 (25%) [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-5.00 sec 0.00 Bytes 0.00 bits/sec 0.069 ms 3740/15029 (25%)

client# iperf3 -c 192.168.1.2 -i1 -p5001 -u -b 200M/10 -t5 server side: Accepted connection from 192.168.1.1, port 55407 [ 5] local 192.168.1.2 port 5001 connected to 192.168.1.1 port 51473 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-1.01 sec 19.7 MBytes 163 Mbits/sec 0.066 ms 300/2820 (11%) [ 5] 1.01-2.01 sec 21.3 MBytes 179 Mbits/sec 0.075 ms 322/3050 (11%) [ 5] 2.01-3.01 sec 20.5 MBytes 172 Mbits/sec 0.182 ms 428/3050 (14%) [ 5] 3.01-4.00 sec 20.4 MBytes 172 Mbits/sec 0.056 ms 449/3060 (15%) [ 5] 4.00-5.01 sec 17.5 MBytes 146 Mbits/sec 0.053 ms 809/3050 (27%) [ 5] 5.01-5.03 sec 8.00 KBytes 2.40 Mbits/sec 0.057 ms 0/1 (0%) [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-5.03 sec 0.00 Bytes 0.00 bits/sec 0.057 ms 2308/15031 (15%)

client# iperf3 -c 192.168.1.2 -i1 -p5001 -u -b 200M/20 -t5 server side: Accepted connection from 192.168.1.1, port 55408 [ 5] local 192.168.1.2 port 5001 connected to 192.168.1.1 port 43183 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-1.01 sec 16.3 MBytes 136 Mbits/sec 0.089 ms 762/2850 (27%) [ 5] 1.01-2.01 sec 18.9 MBytes 157 Mbits/sec 0.062 ms 637/3050 (21%) [ 5] 2.01-3.01 sec 15.2 MBytes 128 Mbits/sec 0.096 ms 1105/3050 (36%) [ 5] 3.01-4.00 sec 17.0 MBytes 143 Mbits/sec 0.084 ms 875/3050 (29%) [ 5] 4.00-5.01 sec 17.6 MBytes 146 Mbits/sec 0.164 ms 799/3050 (26%) [ 5] 5.01-5.05 sec 336 KBytes 78.8 Mbits/sec 0.285 ms 0/42 (0%) [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-5.05 sec 0.00 Bytes 0.00 bits/sec 0.285 ms 4178/15092 (28%)

Since the results are similar, it may not be the burst mode issue?

davidBar-On commented 3 years ago

It is strange that the bust does not have major effect. Maybe it is related to the high rate of packets loss, but I am not sure.

In any case the difference between iperf2 and iperf3 throughput may also be because of the packets size. It seems that iperf3 sends 8KB packets, while iperf2 sends 1460 bytes packets. I suggest that you will try running both with the same packets size, e.g. 1000 bytes (or 8KB), by setting -l 1000 (or (-l 8K). Since iperf3 tests show that throughput can be above 100Mbps, probably both should be run with -b 200M.

ghost999991 commented 3 years ago

It is strange that the bust does not have major effect. Maybe it is related to the high rate of packets loss, but I am not sure.

In any case the difference between iperf2 and iperf3 throughput may also be because of the packets size. It seems that iperf3 sends 8KB packets, while iperf2 sends 1460 bytes packets. I suggest that you will try running both with the same packets size, e.g. 1000 bytes (or 8KB), by setting -l 1000 (or (-l 8K). Since iperf3 tests show that throughput can be above 100Mbps, probably both should be run with -b 200M.

This time I tried b 200M and b 1100M, both with -l 8k

b = 200M

iperf3 client # iperf3 -c 192.168.1.2 -p5001 -b 200M -l 8k -u server Accepted connection from 192.168.1.1, port 35944 [ 5] local 192.168.1.2 port 5001 connected to 192.168.1.1 port 57717 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-1.00 sec 15.5 MBytes 130 Mbits/sec 0.061 ms 832/2820 (30%) [ 5] 1.00-2.01 sec 15.2 MBytes 126 Mbits/sec 0.057 ms 1112/3052 (36%) [ 5] 2.01-3.00 sec 12.8 MBytes 108 Mbits/sec 0.083 ms 1410/3052 (46%) [ 5] 3.00-4.00 sec 15.9 MBytes 134 Mbits/sec 0.150 ms 1018/3052 (33%) [ 5] 4.00-5.01 sec 16.9 MBytes 140 Mbits/sec 0.066 ms 887/3051 (29%) [ 5] 5.01-6.01 sec 17.1 MBytes 144 Mbits/sec 0.060 ms 865/3052 (28%) [ 5] 6.01-7.00 sec 14.8 MBytes 125 Mbits/sec 0.094 ms 1158/3052 (38%) [ 5] 7.00-8.01 sec 13.0 MBytes 108 Mbits/sec 0.071 ms 1387/3052 (45%) [ 5] 8.01-9.01 sec 15.3 MBytes 129 Mbits/sec 0.060 ms 1090/3051 (36%) [ 5] 9.01-10.01 sec 15.9 MBytes 133 Mbits/sec 0.060 ms 1016/3052 (33%) [ 5] 10.01-10.03 sec 0.00 Bytes 0.00 bits/sec 0.060 ms 0/0 (0%) [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-10.03 sec 0.00 Bytes 0.00 bits/sec 0.060 ms 10775/30286 (36%)

iperf2 Because iperf2 can include more commands at server side, I type the same settings at both side

client # iperf -c 192.168.1.2 -p5001 -b 200M -l 8k -u server# iperf -s -i1 -p5001 -u -l 8k -b 200M [ 3] local 192.168.1.2 port 5001 connected with 192.168.1.1 port 44810 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 3] 0.0- 1.0 sec 23.9 MBytes 200 Mbits/sec 0.031 ms 0/ 3054 (0%) [ 3] 1.0- 2.0 sec 23.8 MBytes 200 Mbits/sec 0.026 ms 0/ 3052 (0%) [ 3] 2.0- 3.0 sec 23.8 MBytes 200 Mbits/sec 0.048 ms 0/ 3052 (0%) [ 3] 3.0- 4.0 sec 23.8 MBytes 200 Mbits/sec 0.039 ms 0/ 3051 (0%) [ 3] 4.0- 5.0 sec 23.8 MBytes 200 Mbits/sec 0.019 ms 0/ 3052 (0%) [ 3] 5.0- 6.0 sec 23.8 MBytes 200 Mbits/sec 0.045 ms 0/ 3052 (0%) [ 3] 6.0- 7.0 sec 23.8 MBytes 200 Mbits/sec 0.029 ms 0/ 3052 (0%) [ 3] 7.0- 8.0 sec 23.8 MBytes 200 Mbits/sec 0.034 ms 0/ 3052 (0%) [ 3] 8.0- 9.0 sec 23.8 MBytes 200 Mbits/sec 0.028 ms 0/ 3052 (0%) [ 3] 0.0-10.0 sec 238 MBytes 200 Mbits/sec 0.049 ms 0/30515 (0%)

b= 1100M

iperf3 client # iperf3 -c 192.168.1.2 -p5001 -b 1100M -l 8k -u server Accepted connection from 192.168.1.1, port 35945 [ 5] local 192.168.1.2 port 5001 connected to 192.168.1.1 port 44222 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-1.00 sec 90.1 MBytes 756 Mbits/sec 0.049 ms 35/11569 (0.3%) [ 5] 1.00-2.00 sec 105 MBytes 882 Mbits/sec 0.052 ms 0/13457 (0%) [ 5] 2.00-3.00 sec 107 MBytes 897 Mbits/sec 0.059 ms 11/13699 (0.08%) [ 5] 3.00-4.00 sec 104 MBytes 874 Mbits/sec 0.064 ms 10/13346 (0.075%) [ 5] 4.00-5.00 sec 104 MBytes 874 Mbits/sec 0.053 ms 7/13343 (0.052%) [ 5] 5.00-6.00 sec 109 MBytes 912 Mbits/sec 0.059 ms 0/13919 (0%) [ 5] 6.00-7.00 sec 107 MBytes 897 Mbits/sec 0.056 ms 17/13702 (0.12%) iperf3: OUT OF ORDER - incoming packet = 33632 and received packet = 99167 AND SP = 5 [ 5] 7.00-8.00 sec 104 MBytes 875 Mbits/sec 0.047 ms 1/13354 (0.0075%) [ 5] 8.00-9.00 sec 107 MBytes 895 Mbits/sec 0.060 ms 5/13663 (0.037%) [ 5] 9.00-10.00 sec 104 MBytes 873 Mbits/sec 0.048 ms 0/13328 (0%) [ 5] 10.00-10.03 sec 3.17 MBytes 870 Mbits/sec 0.063 ms 0/406 (0%) [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 5] 0.00-10.03 sec 0.00 Bytes 0.00 bits/sec 0.063 ms 86/133786 (0.064%) [SUM] 0.0-10.0 sec 1 datagrams received out-of-order

iperf2 client # iperf -c 192.168.1.2 -p5001 -b 1100M -l 8k -u server # iperf -s -i1 -p5001 -u -l 8k -b 1100M Server listening on UDP port 5001 Receiving 8192 byte datagrams UDP buffer size: 208 KByte (default) [ 3] local 192.168.1.2 port 5001 connected with 192.168.1.1 port 33735 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 3] 0.0- 1.0 sec 112 MBytes 939 Mbits/sec 0.088 ms 0/14332 (0%) [ 3] 1.0- 2.0 sec 112 MBytes 941 Mbits/sec 0.076 ms 0/14363 (0%) [ 3] 2.0- 3.0 sec 112 MBytes 941 Mbits/sec 0.084 ms 0/14359 (0%) [ 3] 3.0- 4.0 sec 112 MBytes 940 Mbits/sec 0.090 ms 0/14346 (0%) [ 3] 4.0- 5.0 sec 112 MBytes 940 Mbits/sec 0.092 ms 0/14342 (0%) [ 3] 5.0- 6.0 sec 112 MBytes 942 Mbits/sec 0.065 ms 0/14378 (0%) [ 3] 6.0- 7.0 sec 112 MBytes 943 Mbits/sec 0.077 ms 0/14386 (0%) [ 3] 7.0- 8.0 sec 112 MBytes 937 Mbits/sec 0.073 ms 23/14324 (0.16%) [ 3] 8.0- 9.0 sec 78.1 MBytes 655 Mbits/sec 1114.046 ms 0/10000 (0%) [ 3] 9.0-10.0 sec 0.00 Bytes 0.00 bits/sec 0.000 ms 0/ 0 (0%) [ 3] 10.0-11.0 sec 0.00 Bytes 0.00 bits/sec 0.000 ms 0/ 0 (0%) [ 3] 11.0-12.0 sec 0.00 Bytes 0.00 bits/sec 0.000 ms 0/ 0 (0%) [ 3] 12.0-13.0 sec 0.00 Bytes 0.00 bits/sec 0.000 ..ms 0/ 0 (0%) .... [ 3] 25.0-26.0 sec 0.00 Bytes 0.00 bits/sec 0.000 ms 0/ 0 (0%)

-> If I set b = 1100M at server side, the results will show many 0 bit after 10 sec (default); if I only set b = 1100M at client side, then it looks fine (it will not show more than 10 sec).

May I ask what cause this scenario?

Maybe the length of the datagram is the issue, since this time b = 1100M can reach 900Mbps !

davidBar-On commented 3 years ago

Maybe the length of the datagram is the issue, since this time b = 1100M can reach 900Mbps !

That very well may be the case. You can try with different packets lengths to to see how it impact the performance (both iper2 and iperf3).

I also suggest to you -V (verbose) when running both iperf versions and both client and server, to have more info, e.g. the packet size used.

However, there seem to be a network or PC performance/overload issue. When running using -b 200M -l 8k, iperf3 had 30%-45% packets loss, and iperf2 had almost 0% packets loss. On the other hand, when running using -b 1100M -l 8k both had almost 0% loss. It seems that something in the network sometimes causes network overload, or in either the client or server PC something overloads the network buffers. Do you agree? Can you guess/assume what that may be?