Question about Transfer-Encoding header on empty responses

[lenkan]

See reference repository here: https://github.com/lenkan/hio-transfer-encoding-test

This repository creates a server with one endpoint that does not return any data. The example is constructed to mimic the setup of the http server in https://github.com/WebOfTrust/keria without any extras.

Start the server, then do curl:

curl --verbose http://localhost:8081

*   Trying 127.0.0.1:8081...
* Connected to localhost (127.0.0.1) port 8081 (#0)
> GET / HTTP/1.1
> Host: localhost:8081
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 204 No Content
< Server: Ioflo WSGI Server
< Date: Wed, 20 Dec 2023 15:32:31 GMT
< Transfer-Encoding: chunked
<
* Excess found: excess = 5 url = / (zero-length body)
* Connection #0 to host localhost left intact

Notice the Transfer-Encoding: chunked header. Is there a reason for this? Also notice curl reporting Excess found: excess = 5. Also, if the server.py is not set up correclty, please give me pointers.

Background

We are experiencing a lot of issues running keria behind a reverse proxy and this is one of the issues found.

[SmithSamuelM]

transfer encoding chunked enables server sent event SSE support which is used for push notifications.

Not sure what the curl excess found is all about. Couldn't find any docs on Curl just Excess Found: did find some on Excess found on non pipelined read. So have to figure out what your problems are more specifically.

[SmithSamuelM]

It might be sending the empty body as an empty chunk which has chunk terminators and curl is not classifying it correctly. Just guessing.

[lenkan]

Problem appears when proxying requests through a node.js proxy using https://www.npmjs.com/package/http-proxy. Some clients seems to choke and never resolve the response completely. Both of these requests "chains" caused issues:

httpie client => proxy => hio server

And

browser client => proxy => hio server

If I remove the Transfer-Encoding header in the proxy, the requests succeed. Interestingly, using curl as the client resolves the response anyway. I can create a reproduction repo for this if interested?

This might not be a hio issue specifically. The reason I opened the issue here is that I could not understand why the Transfer-Encoding header is set when there is no data to be sent. It seems like some http clients chokes on this. Does it make sense to set Transfer-Encoding: chunked only when the server is sending data or using SSE. Then default to not setting it when the server does not respond with any data?

[SmithSamuelM]

The transfer encoding header presense should be configurable so its not there by default and only when actually using it. I agree with your suspicion that clients that are not opening an SSE connection specifically are choking on the transfer encoding.

[SmithSamuelM]

Transfer-Encoding:chunked is an optional feature that every client should support even when not doing a SSE connection. But because it was added late in the http 1.1 spec evolution few client writers especially for simple clients expect to see it so may incorrectly treat it as a problem. It is valid to send an empty chunk with a chunk size of zero which works as a keep-alive for a connection. This obviates the need for long polling and all those other hackish push techniques in http 1 that were obsoleted with chunked transfer encoding in http 1.1. But many clients were not adapted to take advantage of those features.

https://en.wikipedia.org/wiki/Chunked_transfer_encoding

[kentbull]

I am dealing with something similar with failing requests within KERIA (HIO) to an OOBI URL to a DNS name that is behind a proxy (Oracle load balancer) yet the same exact request with cURL succeeds. Whether the 'Transfer-Encoding' header or a different header I'm dealing with a similar issue. I haven't yet narrowed my issue down to one particular header, or the lack thereof.