HttpClient reuse causes "Connection was closed before the full request has been made"

[ghost]

HttpClient will raise a ProtocolError "Connection was closed before the full request has been made" if there is a long delay between two requests to the same server (presumably due to keep-alive expiring).

Example

import httpclient, os
let client = newHttpClient()
discard client.get("https://quetoo-data.s3.amazonaws.com/")
sleep(10000)
discard client.get("https://quetoo-data.s3.amazonaws.com/blah")

(does not error on some other websites, eg. https://nim-lang.org, but the one above is reliable for me)

Current Output

/home/gibson/test.nim(5) test
/home/gibson/.choosenim/toolchains/nim-#devel/lib/pure/httpclient.nim(1090) get
/home/gibson/.choosenim/toolchains/nim-#devel/lib/pure/httpclient.nim(1066) request
/home/gibson/.choosenim/toolchains/nim-#devel/lib/pure/httpclient.nim(1042) request
/home/gibson/.choosenim/toolchains/nim-#devel/lib/pure/httpclient.nim(1022) requestAux
/home/gibson/.choosenim/toolchains/nim-#devel/lib/pure/httpclient.nim(835) parseResponse
/home/gibson/.choosenim/toolchains/nim-#devel/lib/pure/httpclient.nim(308) httpError
Error: unhandled exception: Connection was closed before full request has been made [ProtocolError]

Expected Output

(does not error)

Additional Information

% nim -v
Nim Compiler Version 1.5.1 [Linux: amd64]
Compiled at 2020-12-22
Copyright (c) 2006-2020 by Andreas Rumpf

git hash: 297c8e403d110dd872e070563328f4e0c734cd01
active boot switches: -d:release

(also reproducible on Windows and Linux 1.4.2)

[DanielBiegler]

Yup, still happens. Just ran into it on v1.6.6 - Relevant other issue: #7413

[edwardyeung]

still happens on v1.6.12 .....

[mantielero]

Is there any workaround or alternative library until this gets fixed?

[ghost]

@mantielero judging from the issue it's about keep-alive, so a workaround would be to create a separate httpclient for a new request.

[edwardyeung]

Is there any workaround or alternative library until this gets fixed?

use puppy instead

[ghost]

@edwardyeung puppy doesn't seem to hold connections at all (like httpclient does), so you can have the same by just creating a new httpclient every request.

[mantielero]

In any case, puppy worked for me. Thanks.

[alexisad]

@mantielero judging from the issue it's about keep-alive, so a workaround would be to create a separate httpclient for a new request.

I created separate client with newAsyncHttpClient (because I am using async) Unfortunately, it doesn't help.

[alexisad]

still happens v1.6.14

[agussecur]

@mantielero judging from the issue it's about keep-alive, so a workaround would be to create a separate httpclient for a new request.

I created separate client with newAsyncHttpClient (because I am using async) Unfortunately, it doesn't help.

This obviously works:

import httpclient, os
let client = newHttpClient()
discard client.get("https://quetoo-data.s3.amazonaws.com/")
sleep(10000)
let client2 = newHttpClient()
discard client2.get("https://quetoo-data.s3.amazonaws.com/blah")