OUT OF ORDER during UDP testing from Mac to Linux

[Hansenq]

Hi there,

When I run a UDP test (such as iperf3 -c SERVER --port 52001 -u -l 1) on my Mac to a server running linux, I get a bunch of the following errors:

iperf3: OUT OF ORDER - incoming packet = 0 and received packet = 0 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 0 and received packet = 0 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 0 and received packet = 0 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 0 and received packet = 0 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 0 and received packet = 0 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 0 and received packet = 0 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 0 and received packet = 0 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 0 and received packet = 0 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 0 and received packet = 0 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 0 and received packet = 0 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 0 and received packet = 0 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 0 and received packet = 0 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 0 and received packet = 0 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 0 and received packet = 0 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 0 and received packet = 0 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 0 and received packet = 0 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 0 and received packet = 0 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 0 and received packet = 0 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 0 and received packet = 0 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 0 and received packet = 0 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 0 and received packet = 0 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 0 and received packet = 0 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 0 and received packet = 0 AND SP = 5
iperf3: OUT OF ORDER - incoming packet = 0 and received packet = 0 AND SP = 5

Both machines are running version 3.1.2, and are on the same network (no NAT in between).

If I run the test, but this time between two Linux boxes, the test works perfectly. This error doesn't occur during TCP testing either.

I have no idea why the packet count for the incoming and received packets are all at 0. Any help would be appreciated. Thanks!

Hansen

[Hansenq]

I looked at the UDP packets being sent out via wireshark, and every single packet contains the exact same information shown above. The only thing that differs between packets is the Identification field in the IP header.

[bmah888]

There's a small buglet here, in that UDP tests need to have at least -l 12 to work properly. The reason is that iperf3 stores some packet metadata (sequence number, etc.) in the UDP packet payload and if the packet send length is too short that metadata gets truncated before being sent. We should really force a lower bound if we're doing UDP tests. In the meantime set a -l parameter of at least 12 and it should work (according to my tests on MacOS to MacOS).

TCP doesn't have this issue, and I don't believe SCTP does either, because they don't need to transfer information in the packet.

[Hansenq]

Ah, thanks for the quick response! If I do -l 12 everything works fine.

However, if I run it without the -l flag, the server doesn't receive anything, and I get the following tcpdump -len output:

19:44:38.870744 xx:xx:xx:xx:xx:xx > yy:yy:yy:yy:yy:yy, ethertype IPv4 (0x0800), length 1514: 128.112.94.34.53001 > 128.112.155.172.52001: UDP, bad length 8192 > 1472
19:44:38.870753 xx:xx:xx:xx:xx:xx > yy:yy:yy:yy:yy:yy, ethertype IPv4 (0x0800), length 1514: 128.112.94.34 > 128.112.155.172: ip-proto-17
19:44:38.870756 xx:xx:xx:xx:xx:xx > yy:yy:yy:yy:yy:yy, ethertype IPv4 (0x0800), length 1514: 128.112.94.34 > 128.112.155.172: ip-proto-17
19:44:38.870760 xx:xx:xx:xx:xx:xx > yy:yy:yy:yy:yy:yy, ethertype IPv4 (0x0800), length 1514: 128.112.94.34 > 128.112.155.172: ip-proto-17
19:44:38.870762 xx:xx:xx:xx:xx:xx > yy:yy:yy:yy:yy:yy, ethertype IPv4 (0x0800), length 1514: 128.112.94.34 > 128.112.155.172: ip-proto-17
19:44:38.870764 xx:xx:xx:xx:xx:xx > yy:yy:yy:yy:yy:yy, ethertype IPv4 (0x0800), length 834: 128.112.94.34 > 128.112.155.172: ip-proto-17

I get the following tcpdump if I run it with -l 2K, showing the same issue

19:47:52.471067 xx:xx:xx:xx:xx:xx > yy:yy:yy:yy:yy:yy, ethertype IPv4 (0x0800), length 1514: 128.112.94.34.58625 > 128.112.155.172.52001: UDP, bad length 2048 > 1472
19:47:52.471070 xx:xx:xx:xx:xx:xx > yy:yy:yy:yy:yy:yy, ethertype IPv4 (0x0800), length 610: 128.112.94.34 > 128.112.155.172: ip-proto-17

This might be another tiny buglet too--from my guesses the IP packet length hasn't been updated to the UDP packet length?

[bmah888]

You're requesting that iperf3 send messages longer than (I am guessing) the maximum packet size on your network. In this case, IP will try to fragment and reassemble the packets. I wonder if the receiver is having problems reassembling the fragments? Any chance you can do tcpdumps on both the client (sender) and server (recever) and see if the fragments are making there?

Some middleboxes (like NAT gateways or stateless / stateful firewalls) will sometimes intentionally drop fragments or do some other weirdness to the packet headers that mess up fragmentation and reassembly. Any chance you have something like that along the path between your two hosts?

[bltierney]

@bmah888 : how about not allowing packet size < 12 to avoid the "OUT OF ORDER" error?

[bmah888]

So there are two problems here. One is that of sending packets that are too small. We can guard against that with a code change as @bltierney just pointed out. Actually upon reading the code again it looks like the magic number is 16...maybe because of some code changes I put in to allow large UDP tests.

The other problem is basically IP fragmentation. In iperf 3.1.5 we changed the default UDP send size to reduce the possibility of this happening, although with the -l option you can still specify a UDP send size that requires fragmentation in the IP layer.

[bmah888]

Added a check for the UDP send size in 03224c9. This should be merged to 3.1, after which this issue should be closed due to a lack of feedback on the IP fragmentation problem/question.

[bmah888]

Merged to 3.1. Closing issue.