samsondav / rihanna

Rihanna is a high performance postgres-backed job queue for Elixir
MIT License
439 stars 49 forks source link

Rihanna.enqueue_many ? #103

Closed lstrzebinczyk closed 3 years ago

lstrzebinczyk commented 3 years ago

I am working on a small project, which focuses on processing some data, and I'm building it based on rihanna. One part of the process is a background job, that spawns multiple followup jobs, and, since Rihanna jobs are based on postgres, I am doing that by introducing multiple rows in one query. It's awesome, and it's enabling the project to work in the first place, since this particular operation allows it to work on a cheap host. The thing is, I got it to work by copying rihanna schemas from https://github.com/samsondav/rihanna_ui, and doing

    now = DateTime.utc_now() |> DateTime.truncate(:second)

    jobs_data =
      for entity_id <- entity_ids do
        term = {MyProject.JobModule, [entity_id]}
        [term: term, enqueued_at: now, priority: 30]
      end

    Homepage.Repo.insert_all(RihannaJob, jobs_data)

Do you think this could be supported by Rihanna natively? It really is a killer feature. I tried to make it work through INSERT INTO SELECT statement, that would really be amazing, but I don't think it's possible in current architecture, since each job data is serialized with :erlang.term_to_binary

WDYT?

lstrzebinczyk commented 3 years ago

Closing the ticket due to lack of response.

tpitale commented 3 years ago

PRs welcome. It seems like a reasonable feature to add.