search

zeromq / libzmq

ZeroMQ core engine in C++, implements ZMTP/3.1
https://www.zeromq.org
Mozilla Public License 2.0
9.53k stars 2.34k forks source link

Connection fails due to ephemeral port clash on localhost. #4588

Closed jamesdillonharvey closed 9 months ago

jamesdillonharvey commented 10 months ago

Please use this template for reporting suspected bugs or requests for help.

Issue description

Part 1

If a zeromq socket attempts connection/reconnection to an ephemeral port on the same host that is not yet active (or recovering from being down), as it cycles through ephemeral ports (for src port) it may attempt reconnection to itself (src/dst ip/port match), the result is a failed connection and no further connection attempts.

A good explanation of "tcp self connection" can be found here: https://totozhang.github.io/2016-01-11-tcp-self-connection/

Part 2

If a new connection (different IP/Port) is made from the same socket after the failed tcp connection happens. When disconnect() is called on the original connection it sometimes closes the new connection also. This happens 50% of the time, I have not been able to reliably recreate so we can focus on part 1 for now.

Environment

Minimal test code / Steps to reproduce the issue

  1. The issue has random timing based on the ephemeral range of the host, to force the issue to happen quicker (if you have a dedicated test system) you can reduce the ephemeral range to 1000 ports. 

    # edit port range
sudo vim /etc/sysctl.conf   # I use 38000 - 39000 range
sudo sysctl -p /etc/sysctl.conf .
sysctl net.ipv4.ip_local_port_range

  2. Run the code which try's to connect to a listening system that does not yet exist.

    
const std::string port = "38500" // port is from new ephemeral range
const std::string url = "tcp://localhost:" + port;  // note I use "localhost" for test but it can be any IP as long as its the IP of one of the NICS on this host.
void *context = zmq_ctx_new();
void *socket = zmq_socket (context, ZMQ_SUB);
int rc = zmq_connect (socket, url.c_str());
assert (rc == 0);
while(1) {sleep(1);}  

3. Run tcpdump on the port you have chosen

sudo tcpdump -i any port 38500 -n


4. Wait until the output of tcpdump stops

# What's the actual result? (include assertion message & call stack if applicable)

The connector is retrying with the ephemeral range port as its src port, when the src and dst port match it starts to connect back on itself, because the IP addresses are the same.  At this point the zmq socket monitor reports Connected  then Immediately Disconnected followed by no more retries.

## tcpdump

13:05:28.177503 IP 127.0.0.1.38472 > 127.0.0.1.38500: Flags [S], seq 1724940016, win 65535, options [mss 65495,sackOK,TS val 1933346856 ecr 0,nop,wscale 12], length 0 13:05:28.177513 IP 127.0.0.1.38500 > 127.0.0.1.38472: Flags [R.], seq 0, ack 1724940017, win 0, length 0 13:05:28.305868 IP 127.0.0.1.38474 > 127.0.0.1.38500: Flags [S], seq 3493934799, win 65535, options [mss 65495,sackOK,TS val 1933346984 ecr 0,nop,wscale 12], length 0 13:05:28.305878 IP 127.0.0.1.38500 > 127.0.0.1.38474: Flags [R.], seq 0, ack 3493934800, win 0, length 0 13:05:28.477346 IP 127.0.0.1.38476 > 127.0.0.1.38500: Flags [S], seq 3295786749, win 65535, options [mss 65495,sackOK,TS val 1933347156 ecr 0,nop,wscale 12], length 0 13:05:28.477356 IP 127.0.0.1.38500 > 127.0.0.1.38476: Flags [R.], seq 0, ack 3295786750, win 0, length 0 13:05:28.577683 IP 127.0.0.1.38478 > 127.0.0.1.38500: Flags [S], seq 33109104, win 65535, options [mss 65495,sackOK,TS val 1933347256 ecr 0,nop,wscale 12], length 0 13:05:28.577691 IP 127.0.0.1.38500 > 127.0.0.1.38478: Flags [R.], seq 0, ack 33109105, win 0, length 0 13:05:28.678079 IP 127.0.0.1.38480 > 127.0.0.1.38500: Flags [S], seq 867330950, win 65535, options [mss 65495,sackOK,TS val 1933347356 ecr 0,nop,wscale 12], length 0 13:05:28.678089 IP 127.0.0.1.38500 > 127.0.0.1.38480: Flags [R.], seq 0, ack 867330951, win 0, length 0 13:05:28.830450 IP 127.0.0.1.38482 > 127.0.0.1.38500: Flags [S], seq 1442296147, win 65535, options [mss 65495,sackOK,TS val 1933347509 ecr 0,nop,wscale 12], length 0 13:05:28.830461 IP 127.0.0.1.38500 > 127.0.0.1.38482: Flags [R.], seq 0, ack 1442296148, win 0, length 0 13:05:28.982832 IP 127.0.0.1.38484 > 127.0.0.1.38500: Flags [S], seq 2813427660, win 65535, options [mss 65495,sackOK,TS val 1933347661 ecr 0,nop,wscale 12], length 0 13:05:28.982843 IP 127.0.0.1.38500 > 127.0.0.1.38484: Flags [R.], seq 0, ack 2813427661, win 0, length 0 13:05:29.142272 IP 127.0.0.1.38486 > 127.0.0.1.38500: Flags [S], seq 2036719940, win 65535, options [mss 65495,sackOK,TS val 1933347820 ecr 0,nop,wscale 12], length 0 13:05:29.142283 IP 127.0.0.1.38500 > 127.0.0.1.38486: Flags [R.], seq 0, ack 2036719941, win 0, length 0 13:05:29.267627 IP 127.0.0.1.38488 > 127.0.0.1.38500: Flags [S], seq 233551879, win 65535, options [mss 65495,sackOK,TS val 1933347946 ecr 0,nop,wscale 12], length 0 13:05:29.267637 IP 127.0.0.1.38500 > 127.0.0.1.38488: Flags [R.], seq 0, ack 233551880, win 0, length 0 13:05:29.385017 IP 127.0.0.1.38490 > 127.0.0.1.38500: Flags [S], seq 59533859, win 65535, options [mss 65495,sackOK,TS val 1933348063 ecr 0,nop,wscale 12], length 0 13:05:29.385025 IP 127.0.0.1.38500 > 127.0.0.1.38490: Flags [R.], seq 0, ack 59533860, win 0, length 0 13:05:29.521385 IP 127.0.0.1.38494 > 127.0.0.1.38500: Flags [S], seq 3620573600, win 65535, options [mss 65495,sackOK,TS val 1933348200 ecr 0,nop,wscale 12], length 0 13:05:29.521395 IP 127.0.0.1.38500 > 127.0.0.1.38494: Flags [R.], seq 0, ack 3620573601, win 0, length 0 13:05:29.681704 IP 127.0.0.1.38496 > 127.0.0.1.38500: Flags [S], seq 2328478056, win 65535, options [mss 65495,sackOK,TS val 1933348360 ecr 0,nop,wscale 12], length 0 13:05:29.681713 IP 127.0.0.1.38500 > 127.0.0.1.38496: Flags [R.], seq 0, ack 2328478057, win 0, length 0 13:05:29.844078 IP 127.0.0.1.38498 > 127.0.0.1.38500: Flags [S], seq 2832671509, win 65535, options [mss 65495,sackOK,TS val 1933348522 ecr 0,nop,wscale 12], length 0 13:05:29.844086 IP 127.0.0.1.38500 > 127.0.0.1.38498: Flags [R.], seq 0, ack 2832671510, win 0, length 0

Issue starts here

13:05:30.029505 IP 127.0.0.1.38500 > 127.0.0.1.38500: Flags [S], seq 2584793936, win 65535, options [mss 65495,sackOK,TS val 1933348708 ecr 0,nop,wscale 12], length 0 13:05:30.029515 IP 127.0.0.1.38500 > 127.0.0.1.38500: Flags [S.], seq 2584793936, ack 2584793937, win 65535, options [mss 65495,sackOK,TS val 1933348708 ecr 1933348708,nop,wscale 12], length 0 13:05:30.029519 IP 127.0.0.1.38500 > 127.0.0.1.38500: Flags [.], ack 1, win 32, options [nop,nop,TS val 1933348708 ecr 1933348708], length 0 13:05:30.029654 IP 127.0.0.1.38500 > 127.0.0.1.38500: Flags [P.], seq 1:11, ack 1, win 32, options [nop,nop,TS val 1933348708 ecr 1933348708], length 10 13:05:30.029661 IP 127.0.0.1.38500 > 127.0.0.1.38500: Flags [.], ack 11, win 32, options [nop,nop,TS val 1933348708 ecr 1933348708], length 0 13:05:30.029677 IP 127.0.0.1.38500 > 127.0.0.1.38500: Flags [P.], seq 11:12, ack 11, win 32, options [nop,nop,TS val 1933348708 ecr 1933348708], length 1 13:05:30.029690 IP 127.0.0.1.38500 > 127.0.0.1.38500: Flags [P.], seq 12:65, ack 12, win 32, options [nop,nop,TS val 1933348708 ecr 1933348708], length 53 13:05:30.029753 IP 127.0.0.1.38500 > 127.0.0.1.38500: Flags [P.], seq 65:92, ack 65, win 32, options [nop,nop,TS val 1933348708 ecr 1933348708], length 27 13:05:30.029781 IP 127.0.0.1.38500 > 127.0.0.1.38500: Flags [F.], seq 92, ack 92, win 32, options [nop,nop,TS val 1933348708 ecr 1933348708], length 0 13:05:30.029786 IP 127.0.0.1.38500 > 127.0.0.1.38500: Flags [.], ack 93, win 32, options [nop,nop,TS val 1933348708 ecr 1933348708], length 0


## socket monitor

socket_monitor.cxx:88 SOCKMON |Event:2:Connect-Delayed| socket_monitor.cxx:88 SOCKMON |Event:128:Closed:FD|21| socket_monitor.cxx:88 SOCKMON |Event:4:Connect-Retried:MS|127| socket_monitor.cxx:88 SOCKMON |Event:2:Connect-Delayed| socket_monitor.cxx:88 SOCKMON |Event:128:Closed:FD|21| socket_monitor.cxx:88 SOCKMON |Event:4:Connect-Retried:MS|127|

.. issue happens here..

socket_monitor.cxx:92 SOCKMON |Event:1:Connected:FD:URL|21|tcp://127.0.0.1.38500| socket_monitor.cxx:92 SOCKMON |Event:512:Disconnected:FD|21|

.. no more reconnection attempts ..



# What's the expected result?

## part 1
I would expect the connection attempts to carry on retrying until disconnect() is called.

## part 2
New connections to other IP addresses made on same socket after the issue occuring should not be effected by calling disconnect() on the localhost connection.