iperf3: error - unable to send control message: Bad file descriptor

bharathk-9 commented 2 years ago

ISSUE: iperf3: error - unable to send control message: Bad file descriptor

Hello Team,

I'm running network testing on my two systems using the iperf3 command, where the testing is getting successful in some time and it's failing in some time with the error message "iperf3: error - unable to send control message: Bad file descriptor". I assume the connectivity is the problem so to test the connectivity between both systems, I am running a ping test which is passed. In my next step, I am doing a retry in case of the first attempt failed in the iperf3 test. in my all attempt the results were the same(frequent failure with the error message).

Iperf Version: iperf 3.9 (cJSON 1.7.13) Command used: iperf3 --client 10.0.0.4 --json

Help me to identify the issue and fix them, Thanks in advance.

swlars commented 2 years ago

Hi, we might need a bit more information to figure out your issue.

What operating system are you using?
Are you immediately retrying the test or is there some kind of sleep?
Can you try with a more recent version of iperf3; the latest is 3.10.1.

bharathk-9 commented 2 years ago

Hi @swlars,

Thank you for your response. please find my inline answers.

What operating system are you using? ANS: Its Ubuntu 18.04.
Are you immediately retrying the test or is there some kind of sleep? ANS: As of now I'm giving some sleep time but again it depends on the time the server starts up(which we don't check as of now).
Can you try with a more recent version of iperf3; the latest is 3.10.1. ANS: let me try and give you update.

As per your 2nd question there can be a chance the client start reaching to server as soon as the server starts up, let me try with some check when the server starts and from there ill give some sleep time. But want to know what is the expected sleep time we want to give so that client can reach to server after the server starts up.

Thanks, Bharath K

dhanfocus commented 2 years ago

That message shows up when the port is not available. Check your firewall settings or the port that you are trying to test.

bharathk-9 commented 2 years ago

I'm not mentioning any port along with command also I don't have any firewall to block, as this is in my own system configured as private network.

dhanfocus commented 2 years ago

If you're not mentioning any port with the command, by default it is going to need to use port 5001. Assuming that you are using this on your private network, you need to enable port forwarding on your home router.

A ping test only tests that the host is alive and uses an ICMP protocol, what you need to test is if a port is blocked.

ilham-Hamzah commented 2 years ago

does it support UPNP?

TheRealDJ commented 2 years ago

The default iperf server port is 5201 which must be open on the server's firewall.

brosahay commented 2 years ago

I am facing the same issue.

What operating system are you using?

PRETTY_NAME="Raspbian GNU/Linux 11 (bullseye)"
NAME="Raspbian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"

Are you immediately retrying the test, or is there some kind of sleep?

Nope.

Can you try with a more recent version of iperf3; the latest is 3.10.1.
$ iperf3 --version
iperf 3.9 (cJSON 1.7.13)
Let me try to get the latest.

The default iperf server port is 5201 which must be open on the server's firewall.

The server is working fine when tested from another machine.

Rtoax commented 2 years ago

me too:

$ iperf3 -c 10.253.38.37
iperf3: error - unable to send control message: Bad file descriptor

embermctillhawk commented 2 years ago

Server computer is Ubuntu 20.04.4 LTS, x86-64 Client computer is Windows 10 Enterprise 2016 LTSB, v1607

iperf3 fails after exactly 49 seconds, every time, with iperf3: error - select failed: Bad file descriptor

Running iperf3 3.11 on the Ubuntu machine, built from the tag here on github.

Running the other way around, with the server on windows and the client on ubuntu, no error is experienced, the test can run indefinitely.

TheRealDJ commented 2 years ago

Does Windows 10 do any outbound traffic blocking? I'd assume you needed to add a firewall exception to allow the inbound traffic. Not sure that this is the answer, but does the same issue occur with the Windows firewall disabled (or defender or whatever)?

embermctillhawk commented 2 years ago

Does Windows 10 do any outbound traffic blocking? I'd assume you needed to add a firewall exception to allow the inbound traffic. Not sure that this is the answer, but does the same issue occur with the Windows firewall disabled (or defender or whatever)?

No, running the same test w10e2016 -> w10e2016 doesn't have the same issue. I believe it is a behavior regression post-3.1.3, because all the windows computers are running 3.1.3, because I haven't gotten around to figuring out how to build iperf3 for windows from source.

To clarify, the test runs for 49 seconds. It works perfectly for 49 seconds, and then terminates unexpectedly. The command used on the ubuntu machine (3.11) is iperf3 -s The command used on the windows machine (3.1.3) is iperf3 -c testmachine1 -b 64m -t 0

However, if the following commands are used instead, iperf3 -s iperf3 -c testmachine1 -b 64m -t 80 Then the test runs for exactly 80 seconds as expected.

Again, running the other way around, ubuntu: iperf3 -c testmachine2 -b 64m -t 0 windows: iperf3 -s The test runs indefinitely as expected.

I haven't tried ubuntu-ubuntu yet, working on it now.

embermctillhawk commented 2 years ago

Critical update! Connecting 3.11 to 3.11 works just fine. It is a non-trivial versioning issue;

The only readily available version of iperf3 for Windows is 3.1.3
The most readily available version of iperf3 for Ubuntu is 3.7 (though 3.11 also exhibits this behavior)

If 3.7 or 3.11 is -s and 3.1.3 is -c, attempting to run an indefinite test -t 0 will fail after 49 seconds. If 3.7 or 3.11 is -c and 3.1.3 is -s, attempting to run an indefinite test -t 0 will succeed. If 3.7 or 3.11 is -s and 3.1.3 is -c, attempting to run a limited test -t 80 will not fail after 49 seconds, but will succeed. If 3.7 is -s and 3.11 is -c, attempting to run an indefinite test -t 0 will succeed.

So the trouble is running post-3.1.3 as server and 3.1.3 as client. Different versions can connect to each other, which is probably desirable behavior, indefinite test behavior differs somehow, and we really really really really badly need updated public binaries.

MrCatEsq commented 2 years ago

This may be irrelevant to OPs case, but this message sometimes shown instead of "connection refused" when, for example, destination port blocked by a firewall or iperf3 server even not runned. I saw it when tried to connect from iperf 3.7 (latest Ubuntu LTS) to iperf 3.7 (previous Ubuntu LTS).

davidBar-On commented 2 years ago

.... this message sometimes shown instead of "connection refused" when, for example, destination port blocked by a firewall or iperf3 server even not runned. I saw it when tried to connect from iperf 3.7 ....

This specific issue was probably fixed by PR #1132. The fix is available only starting from version 3.10.

davidBar-On commented 2 years ago

@embermctillhawk, regarding:

The only readily available version of iperf3 for Windows is 3.1.3

Newer iperf3 version for Windows is available here (although not an official iperf3 site - maintained by BudMan).

markus2330 commented 2 years ago

For me a different error message would already have helped. Maybe something along the lines like davidhan1120 commented on Dec 3, 2021, e.g.

the port is not available. Check your firewall settings or the port that you are trying to test.

If you really want to be beginner friendly, maybe also write that the server might not be running on the given host.

In my man page "control message" does not even occur once.

anohren commented 1 year ago

If 3.7 or 3.11 is -s and 3.1.3 is -c, attempting to run an indefinite test -t 0 will fail after 49 seconds. If 3.7 or 3.11 is -c and 3.1.3 is -s, attempting to run an indefinite test -t 0 will succeed. If 3.7 or 3.11 is -s and 3.1.3 is -c, attempting to run a limited test -t 80 will not fail after 49 seconds, but will succeed. If 3.7 is -s and 3.11 is -c, attempting to run an indefinite test -t 0 will succeed.

Just ran into this exact issue. Always fails after 49-50 seconds using iperf3 -c hostname -R -f M -n 5G on Win 10, but never when substituting with -t 60.

davidBar-On commented 1 year ago

~~Note that PR #648 fixed an issue related to -t 0 around version 3.3, so it is better that you will use a newer version (than 3.1.3 and see if the issue still exists.~~ Newer iperf3 version for Windows is available here: https://files.budman.pw/ (not an official site).

anohren commented 1 year ago

Yes, I saw your earlier message and that solved it. Thanks for the link! Even though my issue wasn't with -t 0 or even with -t.

Paraphraser commented 1 year ago

I have just found an excellent way to trigger this problem:

on the target host, run $ iperf -s
on the source host, run $ iperf3 -c x.x.x.x

iperfmatrix

I'm not necessarily saying that "Connection refused" is any more informative than "Bad file descriptor". I'm also not saying that an iperf/iperf3 mismatch explains all the reported cases. It's just something to consider when trying to figure out why you're seeing this message.

davidBar-On commented 1 year ago

The problem probably happens because iperf and iperf3 are not compatible, so iperf (iperf3) server can work only with iperf (iperf3) client.

Paraphraser commented 1 year ago

Yes. In my case I had been staring at "bad file descriptor" for some time thinking "but this worked a few moments ago, why won't it work now?". I kept re-trying the commands at each end, without ever realising I had omitted the "3" at the server end. Googling the error brought me here but the advice was about firewalls and so on. That's all good advice but was not useful in my case.

What would have been useful was something saying "if you [are silly enough to] have both iperf and iperf3 installed, make sure you're using the same at each end."

When I had the "doh!" moment and understood the cause, it seemed reasonable to assume that other people seeing this error message may well have fallen into the same trap. If they Googled and wound up here, a post explaining "command mismatch" as one possible cause might prove helpful.

davidBar-On commented 1 year ago

What would have been useful was something saying "if you [are silly enough to] have both iperf and iperf3 installed, make sure you're using the same at each end."

Submitted PR #1504 with enhanced error messages that hopefully will help.

phil6564 commented 6 months ago

I am getting exactly this and I am definitely using iperf3 on both the client and server end.

I am doing testing across some spine and leaf switches using two test servers running this version, according to the "uname -a" command: 5.15.0-78-generic #85-Ubuntu SMP Fri Jul 7 15:25:09 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

I am using the following commands on the two servers: Sender: iperf3 -c -t 300 10.100.200.24 --timestamps Receiver: iperf3 -s --timestamps

When the VM attached to one particular leaf switch is involved I find I can run tests ok for a while, and it will continue until the end of the test (300 cycles in the example above) then if I attempt to run the test again it will just sit there doing nothing and eventually come back with "iperf3: error - unable to send control message: Bad file descriptor"

I have found that it will then work again the following morning.

This may be something to do with the leaf switch rather than iperf3 so I am open to any suggestions.

Thanks in advance Phil.

Paraphraser commented 6 months ago

What happens if you use the -p option to change to a non-default port?

I've never seen anything like this but, then, I've never tried running iperf3 on a VM guest so I'm wondering if something in what I'll loosely call the "additional complexity" of getting packets to/from a VM guest is holding onto the default port (5201) longer than it should?

I'm thinking that a pattern of failing, then eventually coming good, fits with a timeout firing to release port 5201. A good test would be wait for the situation to recur and then immediately try with a different port.

If switching to a non-default port succeeds you'd at least have a workaround (just increment the port yourself as part of a script) while you try to figure out whether the underlying cause is something in iperf3, or the VM, or some interaction between both.

If switching to a non-default port still fails then it seems to me the more reasonable conclusion would be some problem in iper3.

Either way you'd have additional info to contribute here.

Hope this helps.

Vishal6ji2 commented 6 months ago

9] 47.00-48.00 sec 113 MBytes 951 Mbits/sec 82130
[ 9] 48.00-49.00 sec 113 MBytes 951 Mbits/sec 82130
iperf3: error - select failed: Bad file descriptor

Server listening on 5201

Getting This error how to resolve this issue?

davidBar-On commented 6 months ago

@Vishal6ji2, can you add more information:

What was the client command that caused this error? (You can hide IP addresses / host names.)
Did you get this error from the client or from the server?
Is there any load balancer server (e.g. HAProxy) between the client and the server? If there is, it may be configured to timeout sessions with no traffic, which may affect the control channel between the client and the server.

esnet / iperf