send_req returns {:error, :retry_later}

[borismol]

It happens because find_best_connection returns already closed connection.

Steps to reproduce:

1. Server which sends header connection: close after each response
2. Quickly perform send_req 10 times
3. 11th send_req returns {:error, :retry_later}

[cmullaparthi]

Agreed. But I don't think anything can be done about this issue. If your server is always closing the connection after each response, then you have to spawn a new connection for each request. This can be done in a couple of ways:

1. Set your max_connections to some value which is greater than the number of requests/sec you expect to generate
2. Explicitly spawn a new connection for each request using ibrowse:spawn_woker_process/1,2 and then use the ibrowse:send_req_direct/4,5,6,7 API

Do you have any other suggestion?

[borismol]

It affects all HTTP/0.9 servers too. src/ibrowse_http_client.erl#L1358

is_connection_closing("HTTP/0.9", _)       -> true;
is_connection_closing(_, "close")          -> true;
is_connection_closing("HTTP/1.0", "false") -> true;
is_connection_closing(_, _) -> false.

I'm not sure whether it is correct fix but for my purposes (communicating with chromedriver) I just remove following code src/ibrowse_http_client.erl#L172

handle_call({send_req, _}, _From, #state{is_closing = true} = State) ->
  {reply, {error, connection_closing}, State};

[cmullaparthi]

That can't be the correct fix. What you are doing by removing that function clause is accepting a request and sending it on a connection which the server has indicated it will be closing after the end of response for the request it is already handling.

As I mentioned in my initial response, the way to work around this situation is to explicitly spawn a connection and then use it for sending a request. The load balancing in ibrowse is designed to support pipelining, which doesn't exist in HTTP/1.0.