HTTP Post Request, HTTP/1.0 fallback, produces error due additional response headers

The following situation happened to us:

We are sending XML datastructures to a server useing POST - inside the INDY code the protocol version is set to HTTP 1.0 so far ok (useing the HTTPOption to disable this behavior resolves the problem)

The response contains also some headers that are not 1.0 available like "Transfer-Encoding: chunked" -- I'am not sure if that was specified for HTTP 1.0? But the response from the server isn't chunked the length information before the true content is missing - that leads to errors ("like line length execeeded" if the content has more than 16k bytes)

My question is - if the protocolversion is reduced to HTTP 1.0 - shouldn't the response handling ignore header entries which are not valid for the requested HTTP Protocol version?

Or is it in this case better to forward the problem to the server developers?

Another Issue with that response and the code handling the headers inside TIdHTTP - if headers are repeated only the first value is taken into account - the same server also returns this:

connection: keep-alive
Content-Type: application/xml;charset=UTF-8
Date: Fri, 24 Mar 2023 07:33:49 GMT
Connection: close

Indy believes the connection=keep-alive, .. but the server closes the connection .. like told in the last header line.

An HTTP/1.1 server is allowed to send an HTTP/1.1 response to an HTTP/1.0 request, but it should not rely on the client being able to understand new headers introduced in HTTP/1.1.

Is the server actually sending an HTTP/1.1 response, or is it sending an HTTP/1.0 response? What does the TIdHTTP.Response.ResponseVersion property report?

Regarding the Connection headers, the HTTP/1.0 and HTTP/1.1 specs both say that a header can appear multiple times only if its value is defined as a comma-separated list, and if combining the header values in the order they are given won't change the sematics of the message.

The Connection and Keep-Alive headers were not officially defined in HTTP/1.0, but some HTTP/1.0 implementations did support them, so the HTTP/1.1 spec does describe them in relation to interoperability with HTTP/1.0. Also in HTTP/1.1, the Connection header IS defined as a comma-separated list. So, the server sending 2 separate Connection headers to an HTTP/1.0 client is not technically wrong, but it is rare. But, in this particular case, it is clearly wrong since keep-alive and close directly conflict with each other. So the server admins need to fix that.

In any case, ideally TIdHTTP should process those 2 Connection headers as if they had been sent as a single Connection: keep-alive, close header, and close the connection if the close token is present. However, it does not currently do that, it only looks at the first Connection header and ignores any subsequent Connection headers, as you noted. The only headers that TIdHTTP currently allows to appear multiple times are Set-Cookie, WWW-Authenticate, and Proxy-Authenticate. That will have to be addressed in a future Indy version.

That being said, the HTTP/1.1 spec does say:

An HTTP/1.1 server may also establish persistent connections with HTTP/1.0 clients upon receipt of a Keep-Alive connection token. However, a persistent connection with an HTTP/1.0 client cannot make use of the chunked transfer-coding, and therefore MUST use a Content-Length for marking the ending boundary of each message.

So, the server is clearly flawed when sending the response you have shown, when TIdHTTP sends an HTTP/1.0 request.

Hi, the server responds with an HTTP/1.1 header as I far I understand this tells the client only that server can also speak 1.1 - but the rest of the response shall be 1.0 combatible?

I'am not sure if the RFC says, the response shall or must not contain any header values which a 1.0 client can't understand?

If it is a shall in the RFC - I think if TidHTTP switches back to 1.0 for the "post" operation - it should parse also the response at the 1.0 level and ignore every 1.1 header value - like Transfer-Encoding and so on. If in the RFC it is a must then the failure is clear on the server side - ArcGIS (ESRI) / Tomcat in this case - returning header entries which are not defined at 1.0 specification.

Currently I can work with the HTTPOptions = [hoKeepOrigProtocol] to disable the HTTP/1.0 fallback this produces a valid response. Is this fallback to HTTP 1.0 still required in the year 2022? Are still there servers online which can't handle a HTTP/1.1 POST command in a Indy compatible manner?

-- the full response header --

HTTP/1.1 200 
Strict-Transport-Security: max-age=31536000
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Transfer-Encoding: chunked
referer: 4037cad9-e2ed-40d4-984b-8162bf5a4d28
referrer: 4037cad9-e2ed-40d4-984b-8162bf5a4d28
response_content_type: application/xml
response_character_encoding: UTF-8
response_buffer_size: 8192
response_status_code: 200
cache-control: no-cache
pragma: no-cache
user-agent: Java/11.0.11
host: host.xyz.de:6443
accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2
connection: keep-alive
Content-Type: application/xml;charset=UTF-8
Date: Fri, 24 Mar 2023 07:33:49 GMT
Connection: close
Server:  

<?xml version="1.0" encoding="utf-8" ?>
<wfs:FeatureCollection xmlns:xs="http://www.w3.org/2001/XMLSchema"> 
....
</wfs:FeatureCollection>

I'am not sure if the RFC says, the response shall or must not contain any header values which a 1.0 client can't understand?

All I can find is this blurb from RFC 2145 Section 2.2:

2.2 Compatibility between minor versions of the same major version

An implementation of HTTP/x.b sending a message to a recipient whose version is known to be HTTP/x.a, a < b, MAY send a header that is not defined in the specification for HTTP/x.a. For example, an HTTP/1.1 server may send a "Cache-control" header to an HTTP/1.0 client; this may be useful if the immediate recipient is an HTTP/1.0 proxy, but the ultimate recipient is an HTTP/1.1 client.

An implementation of HTTP/x.b sending a message to a recipient whose version is known to be HTTP/x.a, a < b, MUST NOT depend on the recipient understanding a header not defined in the specification for HTTP/x.a. For example, HTTP/1.0 clients cannot be expected to understand chunked encodings, and so an HTTP/1.1 server must never send "Transfer-Encoding: chunked" in response to an HTTP/1.0 request.

If it is a shall in the RFC - I think if TidHTTP switches back to 1.0 for the "post" operation - it should parse also the response at the 1.0 level and ignore every 1.1 header value - like Transfer-Encoding and so on. If in the RFC it is a must then the failure is clear on the server side - ArcGIS (ESRI) / Tomcat in this case - returning header entries which are not defined at 1.0 specification.

As pointed out earlier, there are clear faults in the server response anyway, so the server admins should look into fixing them.

Currently I can work with the HTTPOptions = [hoKeepOrigProtocol] to disable the HTTP/1.0 fallback this produces a valid response. Is this fallback to HTTP 1.0 still required in the year 2022? Are still there servers online which can't handle a HTTP/1.1 POST command in a Indy compatible manner?

I don't know. Probably not. Who knows.

IndySockets / Indy

HTTP Post Request, HTTP/1.0 fallback, produces error due additional response headers #453