nbolyard commented 3 years ago

Context

Version of iperf3: 3.6
Hardware: raspberry pi 3 b
Operating system (and distribution, if any): Debian Linux (default pi OS)

Bug Report

I'm trying to run iperf3.6 over a 100kbps link to measuer the actual throughput. I'm running it on two pis. On the server, I run "iperf3 -s --logfile /dev/tty On the client, I run "iperf3 -c (IPv6 address of peer) -u -b 390k -l 1024 -t 12"

Expected Behavior I see 12 reports, one second apart, then the summary
Actual Behavior, on the client I see 11 reports, one second apart, then a pause, then iperf3: error - control socket has closed unexpectedly. On the server I see 16 reports one second apart then iperf3: error - select failed: Bad file descriptor
Steps to Reproduce: given above

Server side:

Server listening on 5201

Accepted connection from (IP addr), port 56688 [ 6] local (ip addr) port 5201 connected to (ip addr) port 54260 [ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 6] 0.00-1.00 sec 9.00 KBytes 73.7 Kbits/sec 37.226 ms 0/9 (0%)
[ 6] 1.00-2.00 sec 10.0 KBytes 81.9 Kbits/sec 55.150 ms 0/10 (0%)
[ 6] 2.00-3.00 sec 11.0 KBytes 90.1 Kbits/sec 65.500 ms 0/11 (0%)
[ 6] 3.00-4.00 sec 9.00 KBytes 73.7 Kbits/sec 72.033 ms 0/9 (0%)
[ 6] 4.00-5.00 sec 11.0 KBytes 90.1 Kbits/sec 73.836 ms 0/11 (0%)
[ 6] 5.00-6.00 sec 10.0 KBytes 81.9 Kbits/sec 74.872 ms 0/10 (0%)
[ 6] 6.00-7.00 sec 11.0 KBytes 90.1 Kbits/sec 74.872 ms 0/11 (0%)
[ 6] 7.00-8.00 sec 10.0 KBytes 81.9 Kbits/sec 75.292 ms 0/10 (0%)
[ 6] 8.00-9.00 sec 10.0 KBytes 81.9 Kbits/sec 75.182 ms 0/10 (0%)
[ 6] 9.00-10.00 sec 11.0 KBytes 90.1 Kbits/sec 75.403 ms 0/11 (0%)
[ 6] 10.00-11.00 sec 10.0 KBytes 81.9 Kbits/sec 89.071 ms 26/36 (72%)
[ 6] 11.00-12.00 sec 10.0 KBytes 81.9 Kbits/sec 53.561 ms 38/48 (79%)
[ 6] 12.00-13.00 sec 10.0 KBytes 81.9 Kbits/sec 36.800 ms 36/46 (78%)
[ 6] 13.00-14.00 sec 10.0 KBytes 81.9 Kbits/sec 25.029 ms 35/45 (78%)
[ 6] 14.00-15.00 sec 11.0 KBytes 90.1 Kbits/sec 17.210 ms 40/51 (78%)
[ 6] 15.00-16.00 sec 10.0 KBytes 81.9 Kbits/sec 13.892 ms 36/46 (78%)
iperf3: error - select failed: Bad file descriptor

Server listening on 5201

Client side: Connecting to host (IP addr) port 5201 [ 5] local (IP addr) port 54260 connected to (ip addr) port 5201 [ ID] Interval Transfer Bitrate Total Datagrams [ 5] 0.00-1.00 sec 48.0 KBytes 393 Kbits/sec 48
[ 5] 1.00-2.00 sec 48.0 KBytes 393 Kbits/sec 48
[ 5] 2.00-3.00 sec 47.0 KBytes 385 Kbits/sec 47
[ 5] 3.00-4.00 sec 48.0 KBytes 393 Kbits/sec 48
[ 5] 4.00-5.00 sec 47.0 KBytes 385 Kbits/sec 47
[ 5] 5.00-6.00 sec 48.0 KBytes 393 Kbits/sec 48
[ 5] 6.00-7.00 sec 48.0 KBytes 393 Kbits/sec 48
[ 5] 7.00-8.00 sec 47.0 KBytes 385 Kbits/sec 47
[ 5] 8.00-9.00 sec 48.0 KBytes 393 Kbits/sec 48
[ 5] 9.00-10.00 sec 48.0 KBytes 393 Kbits/sec 48
[ 5] 10.00-11.00 sec 47.0 KBytes 385 Kbits/sec 47
iperf3: error - control socket has closed unexpectedly

nbolyard commented 3 years ago

This is on a private network that uses valid public network addresses. Don't know why part of the output above is in bold.

nbolyard commented 3 years ago

Problem only occurs with -l 256 and above. Finishes normally with -l 128

TheRealDJ commented 3 years ago

What kind of link is it?

I can run just fine using an ODROID-C2. I don't have an RPI 3B. Amlogic S905 Quad Core Cortex-A53 (ODROID C2) vs Broadcom BCM2837B0 Quad Core Cortex-A53 (RPI). I am running Ubuntu 20.04.1 LTS.

# ./iperf36 --client 192.168.X.X --udp --bandwidth 390k -l 1024
Connecting to host 192.168.X.X, port 5201
[  5]   0.00-1.00   sec  48.0 KBytes   393 Kbits/sec  48  
[  5]   1.00-2.00   sec  47.0 KBytes   385 Kbits/sec  47  
[  5]   2.00-3.00   sec  48.0 KBytes   393 Kbits/sec  48  
[  5]   3.00-4.00   sec  48.0 KBytes   393 Kbits/sec  48  
[  5]   4.00-5.00   sec  47.0 KBytes   385 Kbits/sec  47  
[  5]   5.00-6.00   sec  48.0 KBytes   393 Kbits/sec  48  
[  5]   6.00-7.00   sec  47.0 KBytes   385 Kbits/sec  47  
[  5]   7.00-8.00   sec  48.0 KBytes   393 Kbits/sec  48  
[  5]   8.00-9.00   sec  48.0 KBytes   393 Kbits/sec  48  
[  5]   9.00-10.00  sec  48.0 KBytes   393 Kbits/sec  48  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   477 KBytes   391 Kbits/sec  0.000 ms  0/477 (0%)  sender
[  5]   0.00-10.00  sec   477 KBytes   391 Kbits/sec  0.018 ms  0/477 (0%)  receiver

iperf Done.

Same for IPv6 ...

./iperf36 --client SERVER_IPv6_ADDRESS --udp --bandwidth 390k -l 1024
Connecting to host SERVER_IPv6_ADDRESS, port 5201
[  5] local CLIENT_IPv6_ADDRESS port 52823 connected to SERVER_IPv6_ADDRESS port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  48.0 KBytes   393 Kbits/sec  48  
[  5]   1.00-2.00   sec  48.0 KBytes   393 Kbits/sec  48  
[  5]   2.00-3.00   sec  47.0 KBytes   385 Kbits/sec  47  
[  5]   3.00-4.00   sec  48.0 KBytes   393 Kbits/sec  48  
[  5]   4.00-5.00   sec  47.0 KBytes   385 Kbits/sec  47  
[  5]   5.00-6.00   sec  48.0 KBytes   393 Kbits/sec  48  
[  5]   6.00-7.00   sec  48.0 KBytes   393 Kbits/sec  48  
[  5]   7.00-8.00   sec  47.0 KBytes   385 Kbits/sec  47  
[  5]   8.00-9.00   sec  48.0 KBytes   393 Kbits/sec  48  
[  5]   9.00-10.00  sec  48.0 KBytes   393 Kbits/sec  48  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec   477 KBytes   391 Kbits/sec  0.000 ms  0/477 (0%)  sender
[  5]   0.00-10.00  sec   477 KBytes   391 Kbits/sec  0.019 ms  0/477 (0%)  receiver

iperf Done.

davidBar-On commented 3 years ago

Don't know why part of the output above is in bold

A main issue is that while the client sends about 48KB/sec (390Kbits/sec), the server is receiving only 10KB/sec. It seems that the network throughput is only about 100Kbis/sec. The client probably have a send buffer of about 100KB since the first 100KB were properly received by the server.

After the client completed sending, sending of the 100KB buffered packets continue. This is why the server continue to receive data after the client completed the sending. If my calculations are correct it would take 10 seconds (after the first 12 secs) for the server to receive all buffered data.

Can you send the client and server output when -l 128 was used? In principle it should not change much.

Something happened that that caused the error, I am not sure what, but it may help to try a newer version of iperf3. Version 3.6 is quite old and there were some related fixes since.

TheRealDJ commented 3 years ago

Can you try with 3.9? I see this in the release notes for 3.7 ...

"The delay for tearing down the control connection for the default timed tests has been increased, to more gracefully handle high-delay paths (#751/#859)."

May be related.

TheRealDJ commented 3 years ago

Client is an 1GbE ODROID-C2 running Ubuntu 20.04.1 LTS. Server is a 10GbE rack mount server running Ubuntu 20.04.2 LTS.

I think your link could be causing the issue. Can you provide more details on this 100Kbps link?

# ./iperf36 --client SERVER_IPv4_ADDRESS --udp --bandwidth 390k --time 12 --length 128
Connecting to host SERVER_IPv4_ADDRESS, port 5201
[  5] local CLIENT_IPv4_ADDRESS port 58317 connected to SERVER_IPv4_ADDRESS port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  47.6 KBytes   390 Kbits/sec  381  
[  5]   1.00-2.00   sec  47.6 KBytes   390 Kbits/sec  381  
[  5]   2.00-3.00   sec  47.6 KBytes   390 Kbits/sec  381  
[  5]   3.00-4.00   sec  47.6 KBytes   390 Kbits/sec  381  
[  5]   4.00-5.00   sec  47.5 KBytes   389 Kbits/sec  380  
[  5]   5.00-6.00   sec  47.6 KBytes   390 Kbits/sec  381  
[  5]   6.00-7.00   sec  47.6 KBytes   390 Kbits/sec  381  
[  5]   7.00-8.00   sec  47.6 KBytes   390 Kbits/sec  381  
[  5]   8.00-9.00   sec  47.6 KBytes   390 Kbits/sec  381  
[  5]   9.00-10.00  sec  47.6 KBytes   390 Kbits/sec  381  
[  5]  10.00-11.00  sec  47.6 KBytes   390 Kbits/sec  381  
[  5]  11.00-12.00  sec  47.5 KBytes   389 Kbits/sec  380  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-12.00  sec   571 KBytes   390 Kbits/sec  0.000 ms  0/4570 (0%)  sender
[  5]   0.00-12.03  sec   571 KBytes   389 Kbits/sec  0.042 ms  0/4570 (0%)  receiver

# ./iperf36 --server
Server listening on 5201
-----------------------------------------------------------
Accepted connection from CLIENT_IPv4_ADDRESS, port 38364
[  5] local SERVER_IPv4_ADDRESS port 5201 connected to CLIENT_IPv4_ADDRESS port 58317
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-1.00   sec  46.0 KBytes   377 Kbits/sec  0.028 ms  0/368 (0%)  
[  5]   1.00-2.00   sec  47.6 KBytes   390 Kbits/sec  0.038 ms  0/381 (0%)  
[  5]   2.00-3.00   sec  47.6 KBytes   390 Kbits/sec  0.043 ms  0/381 (0%)  
[  5]   3.00-4.00   sec  47.6 KBytes   390 Kbits/sec  0.038 ms  0/381 (0%)  
[  5]   4.00-5.00   sec  47.6 KBytes   390 Kbits/sec  0.043 ms  0/381 (0%)  
[  5]   5.00-6.00   sec  47.6 KBytes   390 Kbits/sec  0.037 ms  0/381 (0%)  
[  5]   6.00-7.00   sec  47.6 KBytes   390 Kbits/sec  0.019 ms  0/381 (0%)  
[  5]   7.00-8.00   sec  47.5 KBytes   389 Kbits/sec  0.035 ms  0/380 (0%)  
[  5]   8.00-9.00   sec  47.6 KBytes   390 Kbits/sec  0.045 ms  0/381 (0%)  
[  5]   9.00-10.00  sec  47.6 KBytes   390 Kbits/sec  0.034 ms  0/381 (0%)  
[  5]  10.00-11.00  sec  47.6 KBytes   390 Kbits/sec  0.036 ms  0/381 (0%)  
[  5]  11.00-12.00  sec  47.6 KBytes   390 Kbits/sec  0.046 ms  0/381 (0%)  
[  5]  12.00-12.03  sec  1.50 KBytes   361 Kbits/sec  0.042 ms  0/12 (0%)  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-12.03  sec   571 KBytes   389 Kbits/sec  0.042 ms  0/4570 (0%)  receiver

# ./iperf36 --client SERVER_IPv6_ADDRESS --udp --bandwidth 390k --time 12 --length 128
Connecting to host SERVER_IPv6_ADDRESS, port 5201
[  5] local CLIENT_IPv6_ADDRESS port 60491 connected to SERVER_IPv6_ADDRESS port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  47.6 KBytes   390 Kbits/sec  381  
[  5]   1.00-2.00   sec  47.6 KBytes   390 Kbits/sec  381  
[  5]   2.00-3.00   sec  47.6 KBytes   390 Kbits/sec  381  
[  5]   3.00-4.00   sec  47.6 KBytes   390 Kbits/sec  381  
[  5]   4.00-5.00   sec  47.5 KBytes   389 Kbits/sec  380  
[  5]   5.00-6.00   sec  47.6 KBytes   390 Kbits/sec  381  
[  5]   6.00-7.00   sec  47.6 KBytes   390 Kbits/sec  381  
[  5]   7.00-8.00   sec  47.6 KBytes   390 Kbits/sec  381  
[  5]   8.00-9.00   sec  47.6 KBytes   390 Kbits/sec  381  
[  5]   9.00-10.00  sec  47.6 KBytes   390 Kbits/sec  381  
[  5]  10.00-11.00  sec  47.6 KBytes   390 Kbits/sec  381  
[  5]  11.00-12.00  sec  47.5 KBytes   389 Kbits/sec  380  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-12.00  sec   571 KBytes   390 Kbits/sec  0.000 ms  0/4570 (0%)  sender
[  5]   0.00-12.04  sec   571 KBytes   389 Kbits/sec  0.014 ms  0/4570 (0%)  receiver

# ./iperf36 --server
Server listening on 5201
-----------------------------------------------------------
Accepted connection from CLIENT_IPv6_ADDRESS, port 55168
[  5] local SERVER_IPv6_ADDRESS port 5201 connected to CLIENT_IPv6_ADDRESS port 60491
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-1.00   sec  45.9 KBytes   376 Kbits/sec  0.030 ms  0/367 (0%)  
[  5]   1.00-2.00   sec  47.6 KBytes   390 Kbits/sec  0.015 ms  0/381 (0%)  
[  5]   2.00-3.00   sec  47.6 KBytes   390 Kbits/sec  0.022 ms  0/381 (0%)  
[  5]   3.00-4.00   sec  47.6 KBytes   390 Kbits/sec  0.031 ms  0/381 (0%)  
[  5]   4.00-5.00   sec  47.6 KBytes   390 Kbits/sec  0.024 ms  0/381 (0%)  
[  5]   5.00-6.00   sec  47.6 KBytes   390 Kbits/sec  0.036 ms  0/381 (0%)  
[  5]   6.00-7.00   sec  47.5 KBytes   389 Kbits/sec  0.030 ms  0/380 (0%)  
[  5]   7.00-8.00   sec  47.6 KBytes   390 Kbits/sec  0.025 ms  0/381 (0%)  
[  5]   8.00-9.00   sec  47.6 KBytes   390 Kbits/sec  0.034 ms  0/381 (0%)  
[  5]   9.00-10.00  sec  47.6 KBytes   390 Kbits/sec  0.014 ms  0/381 (0%)  
[  5]  10.00-11.00  sec  47.6 KBytes   390 Kbits/sec  0.030 ms  0/381 (0%)  
[  5]  11.00-12.00  sec  47.6 KBytes   390 Kbits/sec  0.030 ms  0/381 (0%)  
[  5]  12.00-12.04  sec  1.62 KBytes   366 Kbits/sec  0.014 ms  0/13 (0%)  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-12.04  sec   571 KBytes   389 Kbits/sec  0.014 ms  0/4570 (0%)  receiver

TheRealDJ commented 3 years ago

?

esnet / iperf

iperf3: error - control socket has closed unexpectedly v.3.6 #1149

Context

Bug Report

Server side:

Server listening on 5201

Server listening on 5201