Improve discoverability

[vereis]

Would be very useful to implement APIs such as:

# Return list of all running jobs...
iex(...)> Monarch.list_running()
[
  %Monarch.Job{name: "Backfill My Thing", running?: true, worker_id: ..., worker: %Oban.Job{...}},
  ...
]

# Would be good to bring in `Ecto.Model` so that we can support `Queryable` like so:
iex(...)> Monarch.list_running(name: "Backfill My Thing")
...

iex(...)> Monarch.list_running(worker_id: ...)
...

# Get job status
iex(...)> Monarch.status(Monarch.Job.t())
...

To build on top of this, it would be useful to have some kinda hook/callback/plugin mechanism so we can do things like send slack messages / emails on job starting/completion/failure.

Not sure what this looks like but maybe:

defmodule MyApp.Hooks.Monarch.Slack do
  @behaviour Monarch.Hook

  @impl Monarch.Hook
  def on_start(%Monarch.Job{} = job) do
    MyApp.Slack.send_message(channel: ..., message: """
    Monarch job #{job.name} starting on #{node()}...
    #{if function_exported?(job.module, :on_start, 1), do: job.module.on_start(job)}
    """)
  end

  @impl Monarch.Hook
  def on_error(%Monarch.Job{} = job, stacktrace) do
    MyApp.Slack.send_message(channel: ..., message: """
    Monarch job #{job.name} errored on #{node()}...
    #{inspect(stacktrace, limit: :infinity, pretty: true)}
    #{if function_exported?(job.module, :on_error, 2), do: job.module.on_error(job, stacktrace)}
    """)
  end

  @impl Monarch.Hook
  def on_complete(%Monarch.Job{} = job) do
    MyApp.Slack.send_message(channel: ..., message: """
    Monarch job #{job.name} completed!!
    #{if function_exported?(job.module, :on_complete, 1), do: job.module.on_complete(job)}
    """)
  end
end

defmodule MyDataMigration do
  @behaviour Monarch
  @behaviour Monarch.Hooks

  @impl Monarch
  def hooks, do: [MyApp.Hooks.Monarch.Slack]

  @impl Monarch.Hooks
  def on_complete(_job), do: "YAY!!!"

  @impl Monarch.Hooks
  def on_error(_job, _stacktrace), do:
    "@engineers!!! This is really bad; please manually intervene!!!!"
  ...
end

[vereis]

Perhaps also automatically wrap jobs in Sibyl traces and we'll automatically have telemetry implemented for all jobs too!

This would just require wrapping Monarch.Worker with @decorate_all trace().

If we wanted to go above and beyond, instead of having @behaviour Monarch in our backfill implementations and have use Monarch instead, which would inject:

quote do
  use Sibyl

  @behaviour Monarch
  @decorate_all trace()  
end