The TCP backend iteratively tries to bind all the addresses in the
configuration file and stops at the first successful one. This makes
rank determination very easy (just look at the index of the address
which was successfully bound), but leads to a special problem when
multiple processes are running on the same machine: Since we use
SO_REUSEADDR (so that we can immediately reuse sockets currently stuck
in the TIME_WAIT state), it it is possible that two processes running
on the same machine can both bind(2) the same address successfully.
Normally this wouldn't be a problem: Since we also call listen(2) on the
socket before committing to a specific address and SO_REUSEADDR doesn't
allow two listening sockets to exist for the same address (that's what
SO_REUSEPORT is for), only one process will be able to successfully call
listen(2), which guarantees that each address is used by exactly one
process (the other one will close that socket and try the next address).
However, during testing I encountered sporadic failures which I
eventually traced to the scenario described above: It seems that if
two processes first call bind(2) and then both call listen(2) at the
same time it is sometimes possible that neither of them succeeds. I
have written a program [0] which provokes this kind of failure and asked
about whether this could be a race condition in the Linux kernel on the
LKML [1].
Still, regardless if this is a bug in the kernel or not, we have to live
with it. This commit changes the TCP backend to retry a failed listen(2)
once, which in my tests makes the problem go away. The program from
[0] suggests that this problem is very unlikely to occur in the first
place (<<1% of the time) and retrying once is probably enough to make
the issue practically impossible to happen. It's not pretty, but until
I get some feedback from the kernel community it's the best that I can
think of.
The TCP backend iteratively tries to bind all the addresses in the configuration file and stops at the first successful one. This makes rank determination very easy (just look at the index of the address which was successfully bound), but leads to a special problem when multiple processes are running on the same machine: Since we use SO_REUSEADDR (so that we can immediately reuse sockets currently stuck in the TIME_WAIT state), it it is possible that two processes running on the same machine can both bind(2) the same address successfully.
Normally this wouldn't be a problem: Since we also call listen(2) on the socket before committing to a specific address and SO_REUSEADDR doesn't allow two listening sockets to exist for the same address (that's what SO_REUSEPORT is for), only one process will be able to successfully call listen(2), which guarantees that each address is used by exactly one process (the other one will close that socket and try the next address).
However, during testing I encountered sporadic failures which I eventually traced to the scenario described above: It seems that if two processes first call bind(2) and then both call listen(2) at the same time it is sometimes possible that neither of them succeeds. I have written a program [0] which provokes this kind of failure and asked about whether this could be a race condition in the Linux kernel on the LKML [1].
Still, regardless if this is a bug in the kernel or not, we have to live with it. This commit changes the TCP backend to retry a failed listen(2) once, which in my tests makes the problem go away. The program from [0] suggests that this problem is very unlikely to occur in the first place (<<1% of the time) and retrying once is probably enough to make the issue practically impossible to happen. It's not pretty, but until I get some feedback from the kernel community it's the best that I can think of.
[0] https://github.com/AlexanderKurtz/listenrace [1] https://marc.info/?l=linux-kernel&m=152707473501460