nelsonic commented 1 year ago

The last thing we want in our logging library is to have a blocking HTTP request
when sending a log entry to our remote logging service. So we need to research, document & implement the best (simplest or fastest) way of doing this using a background process.

Starting point:

Todo

[ ] Create new experiment for capturing the full journey: https://github.com/dwyl/elixir-http-request-tutorial
[ ] Document all the steps taken comprehensively e.g: dwyl/phoenix-content-negotiation-tutorial
[ ] Decide which HTTP library we will use: https://elixirforum.com/t/mint-vs-finch-vs-gun-vs-tesla-vs-httpoison-etc/38588 e.g: should we just use https://github.com/benoitc/hackney because it's already included in all our Phoenix projects e.g: https://github.com/dwyl/mvp/blob/c19c01823db6e4922696f764edb8f0189677cbc1/mix.lock#L30 Or should we explore one of the newer cooler options like Finch: https://github.com/sneako/finch ? If we're going to send the requests in a non-blocking sub-process how much do we care about "performance"? Surely the network latency will totally dwarf any processing speed boost from using a newer fancier library? 💭 These are the questions we want to answer. ❓ (Hopefully we can get a definitive answer for ourselves and others)

ndrean commented 1 year ago

https://andrealeopardi.com/posts/breakdown-of-http-clients-in-elixir/

ndrean commented 1 year ago

An example: run a fake REST API endpoint https://httpbin.org/ with Docker, and then open an IEX session.

Run a fake REST API endpoint https://httpbin.org/ with Docker:

> docker run -p 80:80 kennethreitz/httpbin

You will see periodic HTTP requests responses.

> iex -S mix

defmodule AsyncHttp.MixProject do
  use Mix.Project

  def project do
    [
      app: :async_http,
      version: "0.1.0",
      elixir: "~> 1.15",
      start_permanent: Mix.env() == :prod,
      deps: deps()
    ]
  end

  # Run "mix help compile.app" to learn about applications.
  def application do
    [
      extra_applications: [:logger],
      mod: {AsyncHttp.Application, []}
    ]
  end

  # Run "mix help deps" to learn about dependencies.
  defp deps do
    [
      {:finch, "~> 0.16.0"},
      {:faker, "~> 0.17.0"},
      {:ex_doc, "~> 0.30.3"}
    ]
  end
end

defmodule AsyncHttp do
  @moduledoc """
  Documentation for `AsyncHttp`.
  """
  require Logger

  @doc """
  Usage library "faker" to generate a list of 10 randoms Encoded64 strings
  """
  def generate_data do
    for _ <- 1..10, do: Faker.Food.spice() |> Base.encode64()
  end

  @doc """
  HTTP get request with query string to a REST API called by the HTTP client "Finch".
 This REST API (<https://httpbin.org>) will respond with the Decoded64  string.
  """
  def fetch(data) do
    Finch.build(:get, "http://localhost/base64/#{data}")
    |> Finch.request(:finch)
  end

  @doc """
  Supervised concurrent task. The Task.Supervisor is started in the Application module

  <https://hexdocs.pm/elixir/Task.Supervisor.html#content>
  <https://hexdocs.pm/elixir/Task.html#async_stream/3>

  ## Example

      iex> AsyncHttp.stream
      [
        ok: "Curry Mild",
        ok: "Tarragon",
        ok: "Celery Seed",
        ok: "Lemon Pepper",
        ok: "Nutmeg Whole",
        ok: "Orange Zest",
        ok: "Peppercorns Black",
        ok: "Garlic Powder",
        ok: "Orange Zest",
        ok: "Balti Stir Fry Mix"
      ]
  """
  def stream do
    Task.Supervisor.async_stream(:task_sup, generate_data(), fn data ->
      case fetch(data) do
        {:ok, %{body: body}} ->
          body

        {:error, reason} ->
          Logger.warning(inspect(reason))
      end
    end)
    |> Enum.into([])
  end

  @doc """
  Supervised unlinked task. If the task fails, the parent won't be affected.

  ## Example
      iex> AsynHttp.async
        ["Chicken Seasoning", "Paella Seasoning", "Garam Masala", "Ajwan Seed",
        "Rose Baie", "Garlic Granules", "Steak Seasoning", "Tagine Seasoning",
        "Fennel Seed", "Spice Charts"
      ]
  """
  def async do
    generate_data()
    |> Enum.map(fn data ->
      Task.Supervisor.async_nolink(:task_sup, fn ->
        case fetch(data) do
          {:ok, %{body: body}} ->
            body

          {:error, _reason} ->
            :error
        end
      end)
      |> Task.await()
    end)
  end
end

defmodule Periodic do
  use GenServer
  require Logger

  @moduledoc """
  This process is started by the `Application` module. It will run periodical HTTP requests.
  The REST API must be started (Docker).
  """
  def start_link(_opts) do
    GenServer.start_link(__MODULE__, %{})
  end

  def init(state) do
    schedule_stream()
    Process.send_after(self(), :async, 5_000)
    {:ok, state}
  end

  def handle_info(:stream, state) do
    schedule_stream()
    {:noreply, state}
  end

  def handle_info(:async, state) do
    schedule_async()
    {:noreply, state}
  end

  defp schedule_async() do
    AsyncHttp.async() |> inspect() |> Logger.info()
    Process.send_after(self(), :async, 10_000)
  end

  defp schedule_stream() do
    AsyncHttp.stream() |> inspect() |> Logger.info()
    Process.send_after(self(), :stream, 10_000)
  end
end

defmodule AsyncHttp.Application do
  use Application

  @impl true
  def start(_type, _args) do
    children = [
      {Finch, name: :finch},
      {Task.Supervisor, name:  :task_sup},
      Periodic
    ]

    opts = [strategy: :one_for_one, name: AsyncHttp.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

ndrean commented 1 year ago

When I see the number of HTTP clients, this makes me think that custom libraries should only be based on :httpc whenever possible since you never know which one the user will prefer, and you don't bring in another dependency. I was thinking of the Login libraries for example.

https://elixirforum.com/t/httpc-cheatsheet/50337

!!! Use charlist !!!

'application/json' <=> ~c"application/json" and ~c"#{url}"

case :httpc.request(:post, {~c"#{url}", [], ~c"application/json", body}, [], []) do
      {:ok, {{_, 200, _}, _headers, body}} ->...

dwyl / logs

Side Quest: How to send an `HTTP` Request in a Background Process? #11

Todo