gen_tcp:recv(...) doesn't return EOF

[metelik]

Version: Erlang/OTP 24 [erts-12.1.3] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit]

We encountered this issue with a number of servers utilising the gen_tcp:recv(...) i.e. web (phoenix), sftp...

Setup: https://ipcisco.com/lesson/eapol-extensible-authentication-protocol-over-lan/ Where: Supplicant = Erlang node Authenticator = EAPoL capable switch Authentication server = FreeRadius 3.20.x

Steps to reproduce:

1. On the erlang node, have the wpa_supplicant positively authenticated on an EAPoL enabled switch's port
2. Launch a server on the erlang node
3. Launch an erlang/elixir client app (using the gen_tcp:recv) that'd connect to erlang node's server
4. Stop the authenticaton server
5. On the erlang node, Force reauthentication i.e. restart wpa_supplicant
6. Wait for negative authentication

From now on no traffic will be passed through the switch including and not limited to TCP RST packets.

7. Kill the client
8. Resume the authentication server
9. Force re-authentication

Result: No error is reported by the recv (please bear in mind that formerly connected client is no longer). Actually all :recv calls were still waiting for the input data. On the other hand, the Linux's recv call returned the EOF: As per https://man7.org/linux/man-pages/man2/recv.2.html "When a stream socket peer has performed an orderly shutdown, the return value will be 0 (the traditional "end-of-file" return).", which allowed for a session's closure.

I think the above-described issue could be reproducible with a firewall blocking the RST packets.

[RaimoNiskanen]

Can you show a tcpdump or equivalent of what the machine sees of the connection?

If you do strange things to a TCP connection it is the Host OS that handles it, and the question is what it experiences in this case. gen_tcp:recv/* in Erlang/OTP only reports what the OS tells it, so the next question is what that is. A system call trace such as from strace can show what the Erlang VM sees.

[metelik]

Surely will provide the tcpdump by 4th of April '22. I predict that when EAPoL cuts of the traffic on the port there'd be nothing visible except for EAP initialisatioin requests. QoD:

tcpdump.all.pcap.zip <-- contains all the traffic (192.168.1.13) and SFTP session is established from 192.168.1.6 (last sftp packet is of no. 65) @ packet no. 80 there's an EAP failure visible and until point of a time marked by packet 100 - there's no network traffic passed through the switch's port to which 192.168.1.13 is connected. The client is getting terminated after packet 80 and before packet 100.

tcpdump.ansi-c-server.pcap.zip <------- a tcpdumped traffic between simplistic ansi-c server and client - same thing as with SFTP - no TCP RST packet seen too. The server however (also using recv but as discribed in the aforementioned man page however detects client being gone:

` while( (read_size = recv(client_sock , client_message , 2000 , 0)) > 0 ) { //Send the message back to client write(client_sock , client_message , strlen(client_message)); }

if(read_size == 0) { // <------- Yes we enter here once switch unlocks the port puts("Client disconnected"); fflush(stdout); } else if(read_size == -1) { perror("recv failed"); } `

tcpdump.port_22.pcap.zip <---- just filtered the port 22 related traffic

.

[RaimoNiskanen]

This is still to vague. There is the whole Phoenix framework in between that might do things such as re-try on received 0 bytes.

We need to know what Erlang/OTP is subjected to.

How does a simplistic Erlang server and client behave if placed in the same situation as your simplistic ANSI-C server and client?

[metelik]

I hooked the debug prints in recv returns in i.e. bifrost for the FTP. The recv doesn't return. The t/o is set to 'infinity' so is the the Linux's recv call. I agree it is a bit vague but have absolutely no idea why the OTP behaves differently from the Linux recv implementation (which is a syscall). I started looking in the underlying implementation of :gen_tcp.recv but stuck there hoping for an expert to take a peek maybe - why no 0 bytes return.

[RaimoNiskanen]

Can you write a small Erlang program that demonstrates the issue, and behaves differently than a small C program, reproduce with a firewall that blocks RST as you suggested, and show how exactly that is done.

If I can be sure how to reproduce it, using only Erlang/OTP and basic system features, then I can debug it.

[metelik]

I can give it a go, although Erlang is not my area of expertise. In our project we tend to shift from Erlang to Elixir as soon as possible.

[RaimoNiskanen]

Well, I guess I could translate a small Elixir program (not using e.g Phoenix) that demonstrates the problem into Erlang to debug...

[metelik]

Nah, I wrote an erlang module. See attached. Plus I attach a simplistic ansi-c client and server:

server-client.tar.gz simpleserver.erl.tar.gz

So simplier steps to reproduce:

1. Launch :erlang.simpleserver.server()
2. Connect client from a remote machine simply ./client
3. Disable all incoming traffic on port 8888 i.e.: iptables -A INPUT -i eth0 -p tcp --destination-port 8888 -j DROP
4. Kill the client
5. Re-enable traffic on the 8888 port i.e. iptables -D INPUT 1 (./client's recv returns with 0 bytes whereas the simpleserver.server()'s recv still waits

Double-checked and both actually behave same. So it would not be possible really to reproduce with just iptables manipulation. So the initially described behaviour would be EAPoL specific.

[RaimoNiskanen]

Well. This is the kind of program that I / the OTP group would be inclined to say that it should behave exactly like the corresponding C program, so no surprise there...

The question is then, i f the Phoenix framework sets odd options, uses active mode, has a bug and manages to misplace a message, etc. And exactly how this EAPoL thingie behaves.

Was any of those TCP dumps from the misbehaving server's interface, i.e can we see what the misbehaving server sees of the network traffic. Both TCP dumps show that a while after EAP failure, the server starts ARP:ing for the client's IP address, so the server has apparently realized that it doesn't know it, although there was a connection from the client to the server up before the EAP failure. So is the TCP dump taken from the server's external interface so it shows the server's network stack's point of view?

[metelik]

Apologies, for replying with the delay - I took a bit of a time off. I tried it not only with pheonix but also bifrost (ref. https://github.com/ryancrum/bifrost/blob/c91a176be39535540aa688a7ad24d5d7c7fce479/src/bifrost.erl#L137) so something way simplier. So no weird socket flags. I grabbed tcpdumps from the server side. It looks like the ansi-C recv can somehow catch that something and return w/ 0 which erlang's implementation (laso based on the recv) (https://github.com/erlang/otp/blob/96d598f0e0e10559ca3fb21c78ed88360cff8ab2/erts/emulator/drivers/common/inet_drv.c#L10471) can't. I just wonder how to debug it.. I am thinking on adding debug prints to the kernel's syscall implementation.

[RaimoNiskanen]

Well gen_tcp:recv/*, via inet_drv.c, simply calls libc:s recv(). The interesting interface to observe, I'd guess is the Erlang VM (unix process) - libc interface, so truss/systrace/ktrace...

[RaimoNiskanen]

Cannot reproduce. Inactivity. Feel free to reopen if there is any new info