benoitc / hackney

simple HTTP client in Erlang
Other
1.34k stars 427 forks source link

intermittent {:error, :closed} response #630

Open spencerdcarlson opened 4 years ago

spencerdcarlson commented 4 years ago

I'm not sure if this is the best place to post this. My issues looks like it is similar to 439 but it might just be related to how I am using hackney.

We are making about 4.19K requests per min and seeing a {:error, :closed} response about once every 30 min.

Here is my basic configuration

# create a custom connection pool
defmodule MyApp.Application do
  use Application

  @moduledoc false

  def start(_type, _args) do
    children = [
      :hackney_pool.child_spec(MyApp.ConnectionPool, timeout: 150_000, max_connections: 50)
    ]

    opts = [strategy: :one_for_one, name: MyApp.Supervisor]
    Supervisor.start_link(children, opts)
  end
end
def post do
  url = "https://httpstat.us/200"
  json = Jason.encode!(%{cmd: "do it"})
  :hackney.request(
      :post,
      url,
      [{"content-type", "application/json"}],
      json,
      pool: MyApp.ConnectionPool
    )
    |> handle_response()
end

defp handle_response(response) do
    case response do
      {:ok, code, _, _ref} when is_number(code) and code >= 200 and code < 300 ->
        {:ok, :success}

      {:ok, 401, _, _ref} ->
        {:error, :unauthorized}

      {:ok, status, _, _} ->
        {:error, status}

      error = {:error, _type} ->
        error

      other ->
        other
    end
  end

The majority of the requests are returning successfully, but once every 30 min I am seeing {:error, :closed}

ashish060211 commented 3 years ago

I am also facing this issue. It is intermittent and can be noticed in logs , every hour. Its count is ~3 % of the total request.

alejom99 commented 3 years ago

Hello, we're seeing the same problem. The problem goes away if we disable connection pooling. I should also add that we're using Hackney 1.17.4.

fireproofsocks commented 2 years ago

Same issue with hackney v1.18.0 on Erlang 24.2.1: connections are periodically closed. Setting pool: false seems to fix the issue.

benoitc commented 2 years ago

It is expected that the server close sometimes the connextion. When do you get this error? Is this in the misdle of a request , atthe beginning or at the end ?

fireproofsocks commented 2 years ago

I am triggering periodic API calls via a GenServer that uses Process.send_after to hit an API endpoint every 60 seconds. The failures always seem to happen in minute 3 (on the third request), and the {:error, :closed} seems to happen right as the connection is attempted.

Polexy commented 1 year ago

I also have these errors. sometimes we get {closed,timeout} we also use pool and it starts reproducing when we start using httpS for our tests but when we used http it doesn't reproduce

it happens right as the connection is attempted.

felipe-lopez-solaris commented 1 year ago

which implications does the disabled connection pool has? is there any place to read how the connection pool works?