Startup with dynamic tasks problem

[thomas9911]

Hello, we encounter a problem when restarting our service that uses Quantum.

Context: We have a service that loads dynamic jobs in at startup. We run this in Kubernetes on three pods. We implemented a custom storage backend that stores the jobs in Redis. These get loaded in at startup, however all the three pods execute the same jobs, only at startup, the future scheduled jobs only get executed once (as expected).

Is this a known issue? Maybe you can point us in the correct direction on how to solve this?

Maybe something like an on startup callback that announces to the other nodes that needs to sync data/state ?

Looking forward to your response(s)

[thomas9911]

After some digging we figured out that remove the subtract one fixes our problem. Can you remember why this is there in the clock_broadcaster?

  @impl GenStage
  @spec init(opts :: InitOpts.t()) :: {:producer, State.t()}
  def init(%InitOpts{
        debug_logging: debug_logging,
        storage: storage,
        scheduler: scheduler,
        start_time: start_time
      }) do
    start_time =
      scheduler
      |> Module.concat(Storage)
      |> GenServer.whereis()
      |> storage.last_execution_date()
      |> case do
        :unknown -> start_time
        date -> date
      end
      |> NaiveDateTime.truncate(:second)
      # Roll back one second since handle_tick will start at `now + 1`.
      |> NaiveDateTime.add(-1, :second)  # <--- this part

    :timer.send_interval(1000, :tick)

    {:producer,
     %State{
       time: start_time,
       debug_logging: debug_logging,
       remaining_demand: 0
     }}
  end

[maennchen]

@thomas9911 This was part of #449

[gaffneylg]

We also came up against the same issue in tasks starting unexpectedly, commenting out the 1 second clock skew seemed to solve the problem, thanks for the find @thomas9911