collectiveidea / delayed_job

Database based asynchronous priority queue system -- Extracted from Shopify
http://groups.google.com/group/delayed_job
MIT License
4.81k stars 955 forks source link

Exit after a fixed number of jobs executed. #1164

Open bretweinraub opened 2 years ago

bretweinraub commented 2 years ago

Add command line switch (--exit-after) or Environment variable (EXIT_AFTER),which causes work to exit after a specific number of jobs, irrespective of success or failure.

albus522 commented 1 year ago

What is the problem you are trying to solve with this?

bretweinraub commented 1 year ago

This solves a lot of devops issues that arise from using god or monit, for example:

https://github.com/collectiveidea/delayed_job/blob/master/contrib/delayed_job.monitrc

A whole bunch of problems go away:

So our devops labor went from DJ being the biggest pain point in our infra to essentially zero by switching to the cron/launch-script model. Getting rid of monit has been amazing.

albus522 commented 1 year ago

There are many gotchas and issues to what you are doing that I would not generally advise it. But if it is something you want to continue doing, there is a mechanism you can use that can do what you want without monkey patching from directly within your app. Delayed Job has a plugin infrastructure that we need to better document.

Putting this in an initializer should accomplish what you want:

class QuitAfterXJobs < Delayed::Plugin
  callbacks do |lifecycle|
    if ENV["EXIT_AFTER"]
      lifecycle.around(:perform) do |worker, job, &block|
        result = block.call

        @total_jobs_run ||= 0
        @total_jobs_run += 1
        worker.stop if @total_jobs_run > ENV["EXIT_AFTER"].to_i

        result
      end
    end
  end
end

Delayed::Worker.plugins << [QuitAfterXJobs]