Gun cannot detect end of response

[gaynetdinov]

Hello.

I'm trying to use gun in proxy service, i.e. browser sends request to proxy service, proxy service (using gun) sends a request to another service, which responses with chunked transfer encoding. gun should get chunks from that service and stream it to the browser chunk by chunk. That problem is that with such setup gun cannot properly detect end of the response and the browser does not get full response. To better demonstrate this issue I've created two repositories: https://github.com/gaynetdinov/streaming_test and https://github.com/gaynetdinov/streaming_proxy_test. These are two phoenix apps. Proxy service is localhost:4000, it sends requests to localhost:8080 using gun.

The proxy service looks like this

  def index(conn, _params) do
    host = 'localhost'
    port = 8080
    query = '/'

    {:ok, pid} = :gun.open(host, port)
    monitor_ref = Process.monitor(pid)

    {:ok, _protocol} = :gun.await_up(pid)
    ref = :gun.get(pid, query)

    receive do
      {:gun_response, ^pid, ^ref, :fin, 200, headers} ->
        conn = conn
        |> put_resp_headers(headers)
        |> send_resp(200, "")
      {:gun_response, ^pid, ^ref, :nofin, status, headers} ->
        conn = conn
        |> put_resp_headers(headers)
        |> send_chunked(status)

        async_response(conn, pid, ref, monitor_ref);
      {:gun_response, ^pid, ^ref, :fin, _status, _headers} ->
        conn
      {'DOWN', ^monitor_ref, :process, ^pid, _reason} ->
        conn
    end
  end

  defp async_response(conn, pid, ref, monitor_ref) do
    receive do
      {:gun_data, ^pid, ^ref, :fin, data} ->
        case chunk(conn, data) do
          {:ok, conn} ->
            conn
          {:error, _reason} ->
            conn
        end
      {:gun_data, ^pid, ^ref, :nofin, data} ->
        case chunk(conn, data) do
          {:ok, conn} ->
            async_response(conn, pid, ref, monitor_ref)
          {:error, _reason} ->
            conn
        end
      {:DOWN, ^monitor_ref, :process, ^pid, _reason} ->
        conn
    end
  end

https://github.com/gaynetdinov/streaming_proxy_test/blob/master/lib/streaming_proxy_test_web/controllers/page_controller.ex#L4

So gun sends request to the service, which sends response the following way:

  def index(conn, _params) do
    conn = conn
    |> put_resp_content_type("application/json")
    |> send_chunked(200)

    {:ok, conn} = chunk(conn, "{")
    :timer.sleep(100)
    {:ok, conn} = chunk(conn, "\"key1\"")
    :timer.sleep(100)
    {:ok, conn} = chunk(conn, ":")
    :timer.sleep(100)
    {:ok, conn} = chunk(conn, "\"value1\"")
    :timer.sleep(100)
    {:ok, conn} = chunk(conn, ",")
    :timer.sleep(100)
    {:ok, conn} = chunk(conn, "\"key2\"")
    :timer.sleep(100)
    {:ok, conn} = chunk(conn, ":")
    :timer.sleep(100)
    {:ok, conn} = chunk(conn, "\"value2\"")
    :timer.sleep(100)
    {:ok, conn} = chunk(conn, "}")
    :timer.sleep(100)

    conn
  end

https://github.com/gaynetdinov/streaming_test/blob/master/lib/streaming_test_web/controllers/page_controller.ex#L4

If you open localhost:8080 in browser, you would see proper JSON. You could refresh that page many times and always get the proper JSON.

If you open proxy service localhost:4000 you would also get a proper JSON. However, if you remove at least one :timer.sleep(100) from the above, the response from localhost:4000 wouldn't be full and JSON in the body would be not valid. If you remove all occurrences of :timer.sleep and open localhost:4000 again, you would see only the beginning of the response, sometimes only headers, sometimes only 2-3 first characters of the JSON body.

[essen]

Thanks for the detailed report but unfortunately I have no knowledge of Elixir. It would also help if you could reduce the code involved, surely taking Cowboy's chunked_hello_world and making a few tiny changes would be enough to reproduce the problem in an Erlang shell. Otherwise trying to trace the modules from Gun would also help figure things out.

[gaynetdinov]

It seems like this issue exists only in cowboy 1.1.2 + cowlib 1.0.2. I upgraded my test elixir project to current phoenix master, so I could use cowboy 2.2 + cowlib 2.1.0 and cannot reproduce this issue anymore.

[essen]

Good to know, thanks! Guess we can close this.