oban-bg / oban

💎 Robust job processing in Elixir, backed by modern PostgreSQL and SQLite3
https://oban.pro
Apache License 2.0
3.35k stars 311 forks source link

def handle_info({:notification, :leader, %{"lock" missing in oban/peers/postgres.ex #1146

Closed pablopen closed 2 months ago

pablopen commented 2 months ago

Precheck

Environment

Erlang/OTP 26 [erts-14.1.1] [source] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] [jit]

Elixir 1.16.2 (compiled with Erlang/OTP 26)

Current Behavior

I'm getting this warning message constantly in my logs:

Received unexpected message: {:notification, :leader, %{\"lock\" => \"my-host\"}}", source: :oban, module: Oban.Peers.Postgres]
module=Oban.Peers.Postgres pid=<0.1568.0> 

After some investigation, I think this piece of code present in oban/peers/global.ex is missing in oban/peers/postgres.ex:

  def handle_info({:notification, :leader, %{"lock" => node}}, %State{} = state) do
    {:noreply, %{state | leader: node}}
  end

Because its not there, it triggers oban/peers/postgres.ex:143

  def handle_info(message, state) do
    Logger.warning(
      message: "Received unexpected message: #{inspect(message)}",

[...]

Expected Behavior

Not getting a warning message

sorentwo commented 2 months ago

The Postgres peer doesn't send that notification, and only one peer type should be running at a time.

Did you change the peer recently or do you have a mixture of instances running somehow?

pablopen commented 2 months ago

You put me in the clue to solve this, thank you.

Turns out we run 2 apps in an umbrella, one that we use with the Cron plugin configured as:

config :app_a, Oban,
  peer: Oban.Peers.Global,

and another instance that we just use to launch jobs manually:

config :app_b, Oban,
  plugins: [],
  queues: []

So I solved it adding to :app_b

peer: false

This is how I got mixed peers running at the same time

sorentwo commented 2 months ago

For future reference, setting plugins: false will also disable leadership.

I'm looking into disabling on an empty list as well.