Open danwa5 opened 3 years ago
The only workaround today is to use a pool of connections and threads. Faktory does not offer a PUSH operation which takes multiple payloads. Here's sample code using 10 threads to push in parallel:
require 'faktory'
require 'thread'
require 'securerandom'
ids = (1..1000).to_a
q = Queue.new
ids.each_slice(10).to_a.each {|m| q.push([m[0], m[-1]]) }
q.close
threads = []
10.times do |idx|
job = { "jobtype": "SomeWorker", "queue": "default" }
threads << Thread.new do
c = Faktory::Client.new(debug: true)
while args = q.pop
c.push(job.merge("args": q.pop, "jid": SecureRandom.hex(8)))
end
c.close
end
end
threads.each(&:join)
I like to make a feature request for add support for pushing a large batch of jobs to the Faktory server, something similar to what Sidekiq offers with
#push_bulk
.The network latency of enqueueing a job individually is starting to add up when dealing with a large data set.
Sample code
Is there a workaround for something like this for the time being?