Scheduled jobs set by resque-scheduler are not working with ActiveJob in Rails 4.2

[luismadrigal]

I am using resque, and I am attempting to use resque-scheduler to schedule jobs. I have a schedule that get loaded and the scheduler runs, and even looks like it is running the jobs but it doesn't do anything.

resque-scheduler: [INFO] 2014-09-16T01:54:25-07:00: Starting
resque-scheduler: [INFO] 2014-09-16T01:54:25-07:00: Loading Schedule
resque-scheduler: [INFO] 2014-09-16T01:54:25-07:00: Scheduling friends 
resque-scheduler: [INFO] 2014-09-16T01:54:25-07:00: Schedules Loaded
resque-scheduler: [INFO] 2014-09-16T01:54:55-07:00: queueing FriendsJob (friends)

I can enqueue jobs like this and they get processed.

TestJob.enqueue(params[:id])

and I get this output in the worker log

got: (Job{default} | ActiveJob::QueueAdapters::ResqueAdapter::JobWrapper | ["TestJob", "98732ce5-17f7-4da3-9a03-a5d2f8f74e84", "8"])
** [01:24:01 2014-09-16] 54841: resque-1.25.2: Processing default since 1410855841  [ActiveJob::QueueAdapters::ResqueAdapter::JobWrapper]
** [01:24:01 2014-09-16] 54841: Running before_fork hooks with [(Job{default} | ActiveJob::QueueAdapters::ResqueAdapter::JobWrapper | ["TestJob", "98732ce5-17f7-4da3-9a03-a5d2f8f74e84", "8"])]
** [01:24:01 2014-09-16] 54841: resque-1.25.2: Forked 54882 at 1410855841
** [01:24:01 2014-09-16] 54882: Running after_fork hooks with [(Job{default} | ActiveJob::QueueAdapters::ResqueAdapter::JobWrapper | ["TestJob", "98732ce5-17f7-4da3-9a03-a5d2f8f74e84", "8"])]
Hello World!!
** [01:24:01 2014-09-16] 54882: done: (Job{default} | ActiveJob::QueueAdapters::ResqueAdapter::JobWrapper | ["TestJob", "98732ce5-17f7-4da3-9a03-a5d2f8f74e84", "8"])

But when I try to schedule a job, it looks like the are getting enqueue but they well not doing anything.

Here is the schedule.yml

friends:
  every: "30s"
  queue: "friends"
  class: "FriendsJob"
  args: 8
  description: "Friends jobs scheduler"

Here is the output from the scheduled job.

** [01:23:36 2014-09-16] 54841: got: (Job{friends} | FriendsJob | [8])
** [01:23:36 2014-09-16] 54841: resque-1.25.2: Processing friends since 1410855816 [FriendsJob]
** [01:23:36 2014-09-16] 54841: Running before_fork hooks with [(Job{friends} | FriendsJob | [8])]
** [01:23:36 2014-09-16] 54841: resque-1.25.2: Forked 54880 at 1410855816
** [01:23:36 2014-09-16] 54880: Running after_fork hooks with [(Job{friends} | FriendsJob | [8])]
** [01:23:36 2014-09-16] 54880: done: (Job{friends} | FriendsJob | [8])

After reading this http://dev.mikamai.com/post/96343027199/rails-4-2-new-gems-active-job-and-global-id I am suspecting it has something to do with ActiveJob and GlobalId.

Take a look at the difference enqueued

** [01:24:01 2014-09-16] 54841: Running before_fork hooks with [(Job{default} | ActiveJob::QueueAdapters::ResqueAdapter::JobWrapper | ["TestJob", "98732ce5-17f7-4da3-9a03-a5d2f8f74e84", "8"])]

vs scheduled

** [01:23:36 2014-09-16] 54841: Running before_fork hooks with [(Job{friends} | FriendsJob | [8])]

The jobs themselves were generated via

 rails g job <JobName>

They look like this:

class TestJob < ActiveJob::Base
  queue_as :default
  def perform(friend_id)
    friend = Friend.find(friend_id)
    name = friend.name.swapcase
    puts "Hello World!!"
  end
end

class FriendsJob < ActiveJob::Base
  queue_as :friends
  def perform(friend_id)
    friend = Friend.find(friend_id)
    name = friend.name.swapcase
    puts "Hello World!!"
  end
end

After, writing most of this, I've have removed 'require "active_job/railtie" ' and rewritten the jobs in the resque way and now they work as expected. So I think that the problem is with active_job and the way that scheduled jobs get queued.

[seuros]

What version of Resque are you using ?

cc @cristianbica

[luismadrigal]

I am using rescue 1.25.2 and resque-scheduler 3.0.0

[cristianbica]

We haven't thought at recurring jobs so far and we don't support this as none of the adapters support this without an external gem. However this is a very nice feature but I don't think we can make it in time for 4.2. Also I'm not sure it will suitable to be included in rails

On Tue, Sep 16, 2014 at 10:13 PM, Luis Madrigal notifications@github.com wrote:

I am using rescue 1.25.2 and resque-scheduler 3.0.0

Reply to this email directly or view it on GitHub: https://github.com/rails/rails/issues/16933#issuecomment-55797603

[cristianbica]

As we are not currently supporting recurring jobs with ActiveJob we're going to close this. If it's possible to support recurring jobs I'm seeing this as a separate gem or in rails 5. Feature requests and talks around them are usually talked in the mailing list (https://groups.google.com/forum/#!forum/rubyonrails-core). As a solution to your problem @luismadrigal I suggest you use the resque-scheduler way to do recurring job. thanks

[JustinAiken]

As a workaround, I'm queing Resque schedules (with no args) to an ActiveJob like this:

check_goals:
  queue: "goals"
  every: "5m"
  class: "ActiveJob::QueueAdapters::ResqueAdapter::JobWrapper"
  args:
    -
      job_class: CheckGoalJob
      job_id: #{SecureRandom.uuid}
      queue_name: goals
      arguments:
        -
  description: "Checks all checkable goals to see if they're achieved"

[cristianbica]

@JustinAiken that's nice but as JobWrapper is an internal class that might change I would create a dummy class that implements .performs and enqueues an ActiveJob

[whitehat101]

I just used a slight variation on @JustinAiken's workaround successfully.

I might rewrite all my jobs to use Resque directly, for the time being. It would be nice if ActiveJob supported -- or at least played nice with -- some of the popular job schedulers available.

[cmtonkinson]

@whitehat101 it turned out to be easier for me to simply write to Resque directly for now. @JustinAiken's solution gets the job done, but requires more than a little boilerplate which (in my simple use case) seemed excessive.

[ryanwjackson]

@cmtonkinson When you say "write to Resque directly", how are you doing that?

[cmtonkinson]

@ryanwjackson

@cmtonkinson When you say "write to Resque directly", how are you doing that?

Sorry, that's not really clear, is it? What I meant was that in my (very limited use-case) I simply chose not to use resque-scheduler with ActiveJob. I wrote my jobs as per the vanilla Resque API with the nominal config/resque-schedule.yml pattern.

[ryanwjackson]

After much frustration, I ended up going with the wrapper suggested by @JustinAiken. I wrote the following:

class JobWrapper
  def self.perform(args)
    Object.const_get(args['job_class']).perform_later
  end

  def self.wrap(schedule)
    # from: https://github.com/rails/rails/issues/16933#issuecomment-58945932
    schedule = HashWithIndifferentAccess.new(schedule)
    schedule.each do |k, v|
      next unless v[:class] != 'JobWrapper'
      q = v[:queue] || 'default'

      schedule[k] = {
        class: 'JobWrapper',
        description: v[:description],
        queue: q,
        cron: v[:cron],
        args: [{
          job_class: v[:class],
          queue_name: q,
          arguments: v[:arguments]
        }]
      }
    end
  end
end

Which allows you to keep the regular Resque schedule format and do the following in your setup_schedule Rake task:

Resque.schedule = JobWrapper.wrap(YAML.load_file("#{Rails.root}/config/resque_schedule.yml") || {})

Seems to work for me, but lmk if I'm missing something.

[JustinAiken]

@ryanwjackson - I liked your idea so much I'm turning it into a gem: https://github.com/JustinAiken/active_scheduler

[blaskovicz]

Bump?

[jeremiahgithub]

Using rails 5.1.3 Using resque 1.27.4 Using resque-scheduler 4.3.0

lib/tasks/rescue.rake

require 'resque/scheduler/tasks'
require 'resque/tasks'
namespace :resque do
  desc 'resque'
  task setup: :environment do
    require 'resque'
    require 'resque-scheduler'
    ENV['QUEUE'] ||= '*'
  end
end

config/redis.yml

default: &default
  host: 127.0.0.1
  port: 6379
development:
  <<: *default
  db: 0
test:
  <<: *default
  db: 1
production:
  <<: *default
  db: 2
  host: <%= ENV['REDIS_SERVICE_HOST'] %>
  port: <%= ENV['REDIS_SERVICE_PORT'] %>

config/initializers/resque.rb

REDIS_CONFIG = YAML.load(ERB.new(File.read("#{Rails.root}/config/redis.yml")).result)[Rails.env]
Resque.redis = Redis.new(REDIS_CONFIG)

# dynamically change the schedule
Resque::Scheduler.dynamic = true

# resque-scheduler needs to know about jobs unless +queue+ is set
require "#{Rails.root}/app/jobs/execute_active_job.rb"

Resque.schedule = YAML.load_file("#{Rails.root}/config/resque_schedule.yml")
Resque.before_fork = Proc.new { ActiveRecord::Base.establish_connection }

app/jobs/execute_active_job.rb

module ExecuteActiveJob
  @queue = :execute_active_job
  def self.perform(klass, *args)
    klass = Object.const_get(klass)
    args.empty? ? klass.perform_later() : klass.perform_later(*args)
  end
end

app/jobs/test_job.rb

class IFailed < StandardError; end
class TestJob < ApplicationJob
  queue_as :default

  def perform(*args)
    args.empty? ? raise(IFailed, 'i failed with no args') : raise(IFailed, args[0])
  end
end

config/resque_schedule.yml

# TestJob will raise error with first argument
do_test_job_with_args:
  every: 10s
  class: ExecuteActiveJob
  args:
    - TestJob
    - "I am king"
  description: Kicks off test with argument
do_test_job_without_args:
  every: 10s
  class: ExecuteActiveJob
  args: TestJob
  description: Kicks off test without argument

[svetam]

I've managed to force scheduler to schedule jobs through ActiveJob using extension support approach. Answer on StackOverflow.

[swistak]

Since I stumbled upon this issue. There are now gems that do scheduling using rufus_scheduler (same thing resque-schedule uses) on top of active job.

However if like us you have to fit a square peg into round hole (eg. new app into old deploy scripts / monitoring systems), and want to continue using resque-schedule. Then the easiest workaround I've found is:

Part 1:

class ApplicationJob < ActiveJob::Base
  # This is a workaround for the fact that ActiveJob does not work with Resque & Schedule properly.
  def self.inherited(subclass)
    subclass.const_set :Scheduled, Class.new do
      def self.perform; subclass.perform_now; end
    end
  end
end

Resque / Schedule initializer:

 original_schedule = YAML.load_file(File.join(root, 'config', 'resque_schedule.yml'))
 decorated_schedule = original_schedule.map do |name, definition|
    [name, {'class' => "#{name}::Scheduled"}.merge(definition)]
  end.to_h
  Resque.schedule = decorated_schedule