Clarify rules around half-closed TCP connections

[bradfitz]

The HTTP RFCs say nothing about the expectations around half-closed TCP connections.

In practice, I haven't seen any HTTP client in the wild send a request, and then send a FIN (shutdown) while still waiting for the server's response.

Because we haven't see any clients do that, as of Go 1.8, Go's HTTP server is starting to make assumptions that reading an EOF from the client means that the client is no longer interested in the response. (reading EOF being the closest portable approximation to "the client has gone away").

But in https://github.com/golang/go/issues/18527, a user reports that they have an internal HTTP client which does indeed make a half-closed TCP request.

It would be nice if the HTTP RFCs provided guidance as to whether this is allowed or frowned upon.

I would recommend that the RFC suggest that clients SHOULD NOT half-close their TCP connections while awaiting responses. Because nobody else does, empirically, and relying on reading EOF is a useful signal for servers.

/cc @mnot @benburkert

[mcmanus]

I've maintained a server in the past that received the same bug report. As a matter of compliance I decided the client was right - EOF is a way to delimit a message (they were using it to stream request bodies without chunking). But as a matter of practicality I didn't fix the bug because, as you indicate, ignoring EOF in practice means a lot of useless buffering ending in RST or timeouts.

[bradfitz]

@mcmanus, oh, interesting. And disgusting. I hadn't considered EOF being a means to end an HTTP/1.0 POST. Perhaps I can change Go to make an exception for HTTP/1.0 requests with request bodies.

[bradfitz]

@mcmanus, but @badger points out that RFC 1945 (HTTP 1/.0) says:

The presence of an entity body in a request is signaled by the inclusion of a Content-Length header field in the request message headers. HTTP/1.0 requests containing an entity body must include a valid Content-Length header field.

So using a half-closing a TCP connection was never a valid way to signal the end of an entity body.

[wenbozhu]

This is a pretty important question. Today, it's hard to spec out how cancellation works with REST.

I suspect the proposed spec change will break some users. I wonder if it's possible to measure the impact with real traffic, from the server-side, e.g. the success rate of response completion after a half-close is received. I am happy to run some experiments ...

[RataDP]

Here is a example of half-close connection in the wild, Livestatus. Livestatus is a broker for nagios. To use it via sockets you have to make the query, ex. GET hosts and the close the write channel. After this, the Livestatus returns by the half-closed socket the results.

Example in Python

#!/usr/bin/python
#
# Sample program for accessing the Livestatus Module
# from a python program
socket_path = ("localhost", 6557)

import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(socket_path)

# Write command to socket
s.send("GET hosts\n")

# Important: Close sending direction. That way
# the other side knows we are finished.
s.shutdown(socket.SHUT_WR)

# Now read the answer
answer = s.recv(100000000)

# Parse the answer into a table (a list of lists)
table = [ line.split(';') for line in answer.split('\n')[:-1] ]

print table 

The important snippet is this, when closing the Write channel of the socket:

# Important: Close sending direction. That way
# the other side knows we are finished.
s.shutdown(socket.SHUT_WR) 

Are there any workaround for golang?

Source: https://mathias-kettner.de/checkmk_livestatus.html

[kazuho]

@RataDP Interesting. But is that HTTP?

I see examples like the following on the web page. It does not even look like HTTP/0.9, though I am not sure what the proper syntax of 0.9 is.

root@linux# echo "COMMAND [$(date +%s)] START_EXECUTING_SVC_CHECKS" \
     | unixcat /var/lib/nagios/rw/live

[RataDP]

@RataDP Interesting. But is that HTTP?

I see examples like the following on the web page. It does not even look like HTTP/0.9, though I am not sure what the proper syntax of 0.9 is.

root@linux# echo "COMMAND [$(date +%s)] START_EXECUTING_SVC_CHECKS" \
     | unixcat /var/lib/nagios/rw/live

It could be with unix socket or tcp socket. My fail I only read TCP, i did not see the repo is http :s. My bad, sorry, I came from another issue that was speaking about TCP without the http specification..

[jhatala]

Hello, everyone! I would like to make a case for HTTP/1 disallowing half-closed TCP connections from the client.

What I mean by that is that when an HTTP/1 client shuts down their writing end of the connection (when they send a TCP frame with the FIN flag), that the server can safely assume that the client lost interest in the unsent remainder of the response.

On the other hand, if the server is required to support half-closed TCP connections from the client, in other words if the client is allowed to send a FIN after sending its request and it still expects to receive the response, then the server can only assume that the client lost interest when it receives an RST from the client.

To be able to treat an incoming FIN as an indication of the client's loss of interest has advantages for the server:

• The server can free up resources earlier, upon the reception of the FIN, rather than having to wait for an RST a network round-trip time after sending a data packet.
• The savings are most pronounced when dealing with responses that trickle information down slowly as a form of long polling.
• Some stateful firewalls (NAT) between the client and the server drop packets for connections for which they no longer have state, rather than sending back RSTs. The RSTs thus never make it to the server, the server continues trickling down information until the lack of ACKs causes its window to fill and eventually a write to time out. Server resources held on behalf of a client who lost interest are wasted resources.

I believe that there would be little to no disadvantage to the client. Furthermore:

• HTTP/1.1 encourages the reuse of a TCP connection for a subsequent request.
• Request message bodies are framed using Content-Length or Transfer-Encoding: chunked: "The presence of a message body in a request is signaled by a Content-Length or Transfer-Encoding header field." ( https://tools.ietf.org/html/rfc7230#section-3.3 )
• TLS 1.0 explicitly precludes half-closing at its layer: When one end receives a "close_notify" alert, it is required that it "respond with a close_notify alert of its own and close down the connection immediately, discarding any pending writes." ( https://tools.ietf.org/html/rfc2246#section-7.2.1 )

Thank you.

[royfielding]

A half-close has never been an indication that the client isn't interested in the response. It only indicates that the client isn't going to send any future requests on that connection. Since HTTP/1.x is defined to be independent of the transport, I see no reason to specify half-close – what matters is that the request message be complete. A server is not obligated to send a response, regardless.

[bradfitz]

Since HTTP/1.x is defined to be independent of the transport,

Yeah, but TCP is a pretty popular choice. I think it's worth clarifying what this part of TCP means for HTTP.

This bug exists because implementations are disagreeing on what it means. We're looking to a spec for guidance.

[wenbozhu]

It's pretty hopeless at this point. I propose we clarify that half-close means nothing to HTTP/1.x in this spec, i.e. it's not a cancellation. If any http/1->http/2+ proxies send a rst stream when receiving a half-close from the client, it's a bug.

[royfielding]

The problem with "clarifying it for TCP" is that HTTP runs on any transport with connection qualities and finding a word that means half-close for TCP doesn't always translate into some other transport or session-layer's meaning of half-close, but …

I will try to find a way to do that when we get to the whole "what do we mean by a connection" rewrite in the semantics spec.

In any case, regarding the original question: Go should not interpret an EOF on read as implying an EOF on close. Even if we don't see that as common in standard practice, there are probably thousands of deployed C applications that distinguish the two states; they won't work when some poor soul tries to port them to Go, and you'll be reliving this discussion on a regular basis. For example, IIRC, the first conversation I ever had with @mnot (in 2000) was about using half-close to end requests in iCAP. It happens a lot more frequently in private hacks that are merely using the HTTP libraries for some other purpose.

[mcmanus]

in bkk: no strong consensus. mike bishop to see if he can identify clients that are using half close actively. agreed scope here is h1/tcp

[DavidSchinazi]

I don't think there are any clients currently using this, but I think it would make sense to add guidance stating that half-close does not have any semantics, for all versions of HTTP. Having servers not kill half-closed connections may allow innovation down the line, and I don't think giving this guidance is risky

[wtarreau]

I've seen that quite a bit in field with monitoring scripts involving netcat, as well as data transfer tools. It's a bit less common nowadays since wget and curl are everywhere, but the usual stuff used to be this : $ echo -e "GET /status HTTP/1.0\r\n\r\n" | nc $host:$port | grep -q OK $ echo -e "GET /new-acl-file.txt HTTP/1.0\r\n\r\n" | nc $host:$port | sed '1,/^$/d' > acl-file.txt

In haproxy we don't do anything specific with half-close, we only rely on the end of message, unless the admin specifies "option abortonclose" in which case a half-closed client connection will be aborted if the server has not started to respond.

Also there is no real value in killing half-closed connections. Often people who want to do this mix up the end of message and the end of connection and tend to believe that if the connection is not aborted, there will be no more opportunity to do it. But in fact if the client really aborts, the server will receive RST in response to some send() and will be able to detect the close. I would say that :

• half-closed (shutdown(SHUT_WR)) is a graceful shutdown (i.e. "nothing more to say"), signaled on the wire with a TCP FIN and detected on the server as read()==0
• abort (close()) is a real connection abort (i.e. "stop sending this to me"), signaled on the wire with a TCP RST and detected on the server as send()==-1

The only case where you don't know is until you've started to send(), which explains why haproxy uses its option to decide what to do before receiving the server's response.

I'm just realizing that we could even suggest to send a 100-continue in response to a half-closed connection to probe the connection!

[MikeBishop]

What I've heard back from Microsoft folks with access to old emails is that we've seen half-close behavior from:

• .NET 2.0's HTTP implementation
• An IBM Java client, about which we have little additional detail

Obviously, these are old clients with negligible share. However, that also illustrates the risk: old clients will not be updated to comply with a new spec.

[bradfitz]

Go should not interpret an EOF on read as implying an EOF on close.

Go's HTTP package uses EOF from clients to mean "the HTTP client is probably no longer interested in the server's response". We use it to notify callers of the HTTP server who've registered their interest in knowing when the HTTP client is gone (especially one reading a long-polled response, like Server-Sent Events). Notably, we want to know this immediately upon a FIN, without waiting for a write to fail. (We might not have anything to write for some time.)

While we might ideally use OS-specific TCP stats kernel interfaces to distinguish FIN from RST, we support a dozen+ OSes and the basic interface we can rely on is EOF on FIN. There often isn't a better userspace API available to know the TCP state.

I'd rather the HTTP spec say that clients should not half-close TCP connections, as some servers may interpret a half-closed connection as a client that's no longer interested in the response.

Even if we don't see that as common in standard practice, there are probably thousands of deployed C applications that distinguish the two states; they won't work when some poor soul tries to port them to Go, and you'll be reliving this discussion on a regular basis.

Go has a net package that lets you do low-level networking-y things. Anybody porting low-level C could port to that. The concern for this bug is about Go's net/http behavior, not its net package.

[wtarreau]

Hi Brad!

You must not resort to using the OS to distinguish between the two because you don't need to know that. As you say it's system-specific. And until you send anything you're not guaranteed to get an RST anyway.

The problem you're facing is that you're acting as a proxy between the client and the application. You need a way to let the application know that the client indicated that it stops sending, and only the application can decide if it's an indication of end of transfer or of client abort. Sometimes the application will decide that both are equivalent and that's fine. Of course when you get a notification of error via an RST, it's pretty clear and unambigous. If I may suggest, "just" pass a shut_read info in one case that applications are free to interpret as aborts if they want, and an abort or close info when you're certain it's closed. But really any network-based application must be able to tell the difference between half-closed and full-closed otherwise it's deemed to either abusively close on regular half-close or ignore real closes sometimes.

In practice people tend to see TCP as a bidirectional stream while it's in fact two independent unidirectional streams. The only link between the two are the ACK numbers passed along packets to confirm receipt. But FTP servers for example are well used to seing half-closed connections since the data channel can be half-closed during the whole transfer.

[bradfitz]

@wtarreau, Go's HTTP package has supported its CloseNotifier API since 2013-05-13 and the Go compatibility promise means it's not going away. We can improve the implementation, but we can't pretend the problem doesn't exist and remove the feature. It came about because it was frequently requested. Authors of HTTP server handlers want to know when people close tabs in their browser, etc.

[wtarreau]

@bradfitz OK that's perfect. Then it's more a matter of clarifying what each event means and what shortcuts may be taken with what impacts. In practice it's fine most of the time, it's just that it's important to be clear about what this really means. For example it's fine to say "you can reasonably assume that a close notification indicates a closed tab and that you can abort an ongoing operation if your application is designed to work solely with a browser, but a more robust application should consider that it only indicates the client has nothing more to send and that the server must close just after sending the final response ; specifically, some scripts or API clients may induce a close event immediately after sending a request and while waiting for a response".

[mnot]

Discussed in Bangkok; we need to collect data about behaviour.

[Lukasa]

As an extra data point, both SwiftNIO and Netty treat receiving EOF on read as an indication the client is no longer interested in the response. We're definitely not happy with this: from our perspective, clients should be able to send FIN without us giving up on them. I'd be in favour of wording to disincentivise servers from doing what we (currently) do.

[mcmanus]

ietf104: significant interest in documenting _something_here. at least "client should not do this" - hummed the question of discouraging the server from closing connection on rcpt of fin. a decent support for that pr.

[MikeBishop]

There seems to be a really simple hack for differentiating FIN/RST. As @bradfitz noted, a write will fail if a RST was received. This issue only applies to HTTP/1.1. Therefore: upon read-EOF, immediately write the "HTTP/1.1 " of the response line. If that write fails, you can safely abort generation of the rest of the response, because it was a RST.

Or does this not work on certain platforms?

[Lukasa]

There are also various OS level options. Both epoll and kqueue can be used to distinguish FIN and RST, so it’s usually a matter of distinguishing the two cases in higher level APIs.

[wenbozhu]

Good that this issue is still open :)

Since the server always knows what works or not, writing a first-line header in response to FIN does allow most of the servers to detect aborted clients immediately.

Maybe the spec can just clarify that FIN (half-close) should not be used by the server to cancel the request unless the server has a way to detect that the underlying TCP connection has been shutdown (i.e. the peer has gone away).

[mnot]

Reading through the discussion, a couple of notes;

• We already talk about half closed connections on the server side in 9.7 Tear-down, so talking about them on the client side isn't prima facie out of scope.
• It might be helpful to explicitly note that requests are not delimited by half-close in 6.3 Message Body Length. That's not a normative change, just a clarification, and it aligns with broad implementation. It would also satisfy the desire to document "client should not do this" at least in that case.
• It seems like the best we can do on the server side (in terms of interpreting FIN from the client) is to document that some servers assume it to be a loss of interest, but this may not be reliable in all cases, and should not be relied upon by clients.

[mnot]

^ PTAL

[wtarreau]

In 9.6 "tear down", after "If the server receives additional data from the client on a fully closed connection, such as another request that was sent by the client before receiving the server's response", I'd add "or the message body". The RST problem is far more visible with POST requests that servers reject due to authentication, redirects, lack of cookies etc, where servers are tempted to respond and immediately close without draining the entire request message.

Regarding the questions above, I'm still seeing a few possible hints/workarounds for server implementers who really want to have good interoperability and good assurance that a tab was closed. One hint is to emit a "100-continue" interim response in response to a half-closed request if the request was made over HTTP/1.1. If the client is still there, it will have no effect. If the client has completely closed, this one will trigger a reset from the client which will be detected by the server. The other solution is to consider that if a client sent a request in HTTP/1.1 without "connection: close" and finally half-closed the request side, it is extremely unlikely to be a simple implementation and that it can be assumed with good confidence that the client really wants to abort request processing.

[mnot]

Hey Willy,

Would something like this do the trick?

Note that common TCP APIs make it difficult to distinguish a half-closed client from one that is no longer accepting data. In this case, a server that hasn't yet sent a final response can distinguish these cases by sending a 100 (Continue) non-final response.

I'm a bit wary of putting this in the spec, since it's so specific and advisory. What do others think?

[wtarreau]

I agree both with the text and your concern. Maybe we can enforce the fact that it's absolutely not a spec and just a hint by saying "... a server that really needs to distinguish these cases could for example send a 100 (Continue) non-final response if it hasn't yet sent a final response".

This way it's clearly worded as a hint to work around existing API limitations and to remind developers that determining whether a client closed a browser window is not rocket science over TCP.

[wenbozhu]

can distinguish these cases by sending a 100 (Continue) non-final response.

I would implement something like a zero-byte chunk, or a small mount of dummy data (safe to the MIME type such as JSON), which could also help "keep-alive" a streaming response without introducing a non-standard C-T. If a server has already generated the C-L header, I would just let the request run to completion.

Or are we concerned about proxies buffering the response body, because 100 will always reach the client immediately?

[wenbozhu]

BTW, I actually plan to implement some sort of cancellation support soon for web frontends behind google.com; and I could help report some data wrt the detection mechanism, with or without the app responding to the cancellation event.

[wtarreau]

I would implement something like a zero-byte chunk, or a small mount of dummy data (safe to the MIME type such as JSON), You can't send a zero-byte chunk before headers (and even less data of course). The only other reasonably interoperable thing you could send before headers to probe the connection would be a CRLF since most agents will ignore them before a message. Another approach would be to start sending "HTTP/1.1" without the status yet (or just "HTTP" without even the version), but it can be trickier as it will require to remember that this part was already emitted.

Now after the headers it's different, you're already sending response data so you don't care, you'll know very soon if the client is still there.

[wenbozhu]

I was wrong. What I proposed (i.e. writing extra body bytes) only works for frameworks that generate headers immediately without waiting for applications to write any data (which might include errors but as application payload).

Re: 100-continue response,

I noticed that the following restriction stated in rfc2616 was removed from rfc7231.

"An origin server SHOULD NOT send a 100 (Continue) response if the request message does not include an Expect request-header field with the "100-continue" expectation. "

Curious what's the background for this change.

[wtarreau]

I suspect the reason was the prevision for new 1xx status codes, that just clients can remap to pure 100 when they don't know them. Note, I proposed 100 but any 1xx (except 101) would fit. For example we might instead send "102 Processing" (rfc2518), which could be even more suitable and possibly less confusing.

[wenbozhu]

For example we might instead send "102 Processing"

I could live with this. Will report back how frequently we are seeing FINs with a pending http/1.1 request on the client side.

It might be helpful to explicitly note that requests are not delimited by half-close in 6.3 Message Body Length.

Yes.