Should error on incomplete transfer with `Transfer-Encoding: chunked`

[ghost]

When streaming content from a server where the response is Transfer-Encoding: chunked, if the transfer is somehow stopped half way. curl will error curl: (18) transfer closed with outstanding read data remaining with 18 as the return code, while httpie reports everything is fine with return code 0.

[sigmavirus24]

There's a few places this problem could be bubbling up from (listed in order of worst case to best case):

• httplib doesn't raise any error when the server stops
• urllib3/requests re-raise httplib's exception using an commonly used exception that httpie catches & ignores
• httpie is catching the underlying exception and not setting the proper exit status

Coincidentally, I've also listed these in the order that I consider most likely to least likely. httplib elides the chunked transfer-encoding and presents the data to the user as if it were just a continuous stream of bytes. If you wanted to validate the chunk-length an inordinate amount of digging (to retrieve the underlying socket) would be necessary and you'd have to completely bypass httplib (something urllib3 does not currently do).

I'm not familiar with curl's error codes, but I suspect the error your seeing is due to the server not completely transferring a chunk. If curl is raising the same error for that case as for the case where it doesn't receive the second CRLF to end the response body, then that's really annoying and a very bad user experience (because how is one supposed to know if all the data finished transferring but the server closed the connection before sending the final CRLF vs the connection closed midway through a chunk?). That said, this will be hard to fix without something to test it against. Do you have something you can share @Laec?

[ghost]

In my case a I have an example server that responses "hello" and then error internally instead of returning normally.

I captured the response with wireshark, the error response for the data is basically:

5
hello

If I change the example server not to error, the response for the data is:

5
hello
0

So in my case a chunk (size 5, content "hello") is fully transferred but anything after the error (if it had any) including the terminating 0 was never sent due to the server error.

[sigmavirus24]

Right. I forgot about the terminating 0<CRLF>. I'm still tempted into believing this is something unfixable thanks to the absolute terribleness that is httplib