Supervisor.child_spec/2 raise an error when launching the same Module (process) twice

[smahi]

I am using GenStage to create data processing pipeline. With a single consumer PageConsumer it works fine, but not with multiple processes for the same consumer module PageConsumer, instead it raises the following error.

Environment

• Elixir & Erlang/OTP versions (elixir --version):

Erlang/OTP 24 [erts-12.1] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [jit]

Elixir 1.12.3 (compiled with Erlang/OTP 24)

• Operating system: ubuntu 20.04

Current behavior

defmodule Scrapper.Application do
  @moduledoc false

  use Application

  @impl true
  def start(_type, _args) do
    children = [

      {Scrapper.PageProducer, []},
      Supervisor.child_spec({Scrapper.PageConsumer, []}, id: :consumer_a),
      Supervisor.child_spec({Scrapper.PageConsumer, []}, id: :consumer_b)
    ]

    opts = [strategy: :one_for_one, name: Scrapper.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

$ iex -S mix

Raises the following Error:

Erlang/OTP 24 [erts-12.1] [source] [64-bit] [smp:12:12] [ds:12:12:10] [async-threads:1] [jit]

13:07:30.582 [info]  Page Producer init...

13:07:30.586 [info]  Page Consumer init...

13:07:30.586 [info]  Producer received demand for 1 pages

13:07:30.586 [info]  Application scrapper exited: Scrapper.Application.start(:normal, []) returned an error: shutdown: failed to start child: :consumer_b
    ** (EXIT) already started: #PID<0.197.0>

13:07:30.593 [info]  Application gen_stage exited: :stopped
** (Mix) Could not start application scrapper: Scrapper.Application.start(:normal, []) returned an error: shutdown: failed to start child: :consumer_b
    ** (EXIT) already started: #PID<0.197.0>

Expected behavior

Launch two processes :consumer_a and :consumer_b

[josevalim]

The ID identifies a process under the supervisor but the process may still have its own name, which should also be unique. My guess is that the Scrapper.PageConsumer defines a :name under its start_link that is being used. You most likely want something like this:

      Supervisor.child_spec({Scrapper.PageConsumer, name: :consumer_a}, id: :consumer_a),
      Supervisor.child_spec({Scrapper.PageConsumer, name: :consumer_b}, id: :consumer_b)

If you share the code for PageConsumer I can provide more detailed feedback. :)

[smahi]

Thank you so much for your help @josevalim This is the code of consumer module

defmodule Scrapper.PageConsumer do
  use GenStage

  require Logger

  alias Scrapper.PageProducer

  def start_link(_args) do
    GenStage.start_link(__MODULE__, [], name: __MODULE__)
  end

  def init(initial_state) do
    Logger.info("Page Consumer init...")

    {:consumer, initial_state, subscribe_to: [{PageProducer, min_demand: 0, max_demand: 1}]}
  end

  def handle_events(events, _from, state) do
    Logger.info("Page Consumer received #{inspect(events)}")

    events
    |> Enum.each(fn _page ->
      Scrapper.work()
    end)

    {:noreply, [], state}
  end
end

[josevalim]

Yes, the issueis here:

  def start_link(_args) do
    GenStage.start_link(__MODULE__, [], name: __MODULE__)
  end

Do this:

  def start_link(opts) do
    GenStage.start_link(__MODULE__, [], name: opts[:name])
  end

And change the supervisor code to the one in the previous comment and you should be good!

[smahi]

@josevalim It worked like charm. By the way I am reading a book called Conccurent Data Processing by Svilen Gospodinov published by The Pragmatic Bookshelf, I took my code from the book examples.

Thank you so much @josevalim