Jobs won't start until ActiveRecord loaded - gotcha in development environment

[isaac]

Hi 👋

Just thought I would post this here because it had me stumped for a while - I thought about updating the docs but wasn't sure where it should go 🤷

If your Rails app is running in development mode with the following settings config.good_job.execution_mode = :async and config.eager_load = false (both are the defaults I believe), when you start your rails app no enqueued jobs will be started until after ActiveRecord is loaded.

I first ran into this issue by starting my Rails server and trying to enqueue a job from the Rails console. The job enqueued ok but never ran until I ran at least one web request. The same thing happens if you have previously enqueued jobs when you start your Rails server. This happens because if config.eager_load = false then ActiveRecord is lazy loaded - the web request would trigger the loading. GoodJob won't start any async jobs until after ActiveRecord is loaded: https://github.com/bensheldon/good_job/blob/ec385cf6b1890bf1b5ff635f77360ace733d8689/lib/good_job/dependencies.rb#L36

The fix for this issue is to set config.eager_load = false in development.rb

Hope this is helpful to future users - if you want me to add this to the docs just let me know where.

Some details about my environment below.

OS: macOS Ventura 13.3.1 Rails: 6.1.6 Ruby 3.1.2 GoodJob 3.15.0

Cheers, Isaac

[bensheldon]

Sorry that you ran into that rough edge!

The current behavior of not autoloading intentionally makes Rails boot as fast as possible in Development. Directionally in Rails, we're trying to decouple anything that might trigger eager autoload e.g. https://github.com/rails/rails/issues/45434

The following is the smallest footprint way to ensure that GoodJob async will start at boot in Development when eager_load is false:

# config/initializers/eager_load_good_job.rb

if Rails.env.development?
  # Eager autoload Active Record and Active Job so that
  # GoodJob will be started during application boot.
  Rails.application.config.after_initialize do
    ActiveRecord::Base
    ActiveJob::Base
  end
end

[bensheldon]

In terms of documentation, I don't really want to recommend that people use the workaround. It's more like "fyi, GoodJob won't start until Active Record and Active Job are loaded, which may not happen until the first web request when Rails' config.eager_load = false"

I'm curious whether, when you were in the Rails console, running GoodJob.perform_inline after enqueueing your job would have been sufficient. (I'm not really designing for multi-process orchestration in Development).

[isaac]

@bensheldon thanks for getting back to me so fast!

I can see why you wouldn't want to recommend that workaround, but I can also see where it might be useful to know about in specific situations. For my purposes setting config.eager_load = false was sufficient.

Using GoodJob.perform_inline would have solved the technical issue of actually running the job, but that was not my goal. I'll take a step back and explain in more detail why I was trying to run a job in this particular way.

I've used Sidekiq for years, but for a recent project I wanted to experiment with using GoodJob because the workload didn't justify running a separate worker process. I've also been following the project for a while and have been looking for an excuse to use it :) I added it to my project and then set it up with a simple job. I've never run jobs within the Rails server before, so I wanted to get a sense of how it worked before I went any further. I use the Rails console a lot during development, so it seemed like this would be the easiest way to kick off a job. This method works with Sidekiq out of the box so I was very confused as to why it wasn't working with GoodJob. Running GoodJob.perform_inline would probably have worked (I haven't tried this), but I was trying to get a sense of how running jobs worked within the Rails server, not within the console - the console was just an easy way for me to enqueue the job. I was even more confused when I enqueued a job and then restarted the server and the job still didn't run. Luckily I had some time to dig into what was going on - if I was short of time I may just have given up at this point and gone back to Sidekiq.

So in summary, for me this was less of a technical issue like "how to run a job from the console" and more of a developer experience issue like "why is it so hard to get started with this project". Hope this helps to clarify where I was coming from.

[bensheldon]

@isaac that's such a helpful explanation! Thank you!

If you can remember back to your first-experience, if I added this explanation and code snippet to the Readme, where do you think I should add it and were there any keywords you were CMD-F'ing for so I can make sure it would show up?

Like I'm wondering if the explanation should go under "Getting Started" or somewhere else?

[isaac]

I think it would be helpful if there was at least a link to it in step 4 of the set up: https://github.com/bensheldon/good_job#set-up

But the full explanation probably belongs here: https://github.com/bensheldon/good_job#execute-jobs-async--in-process

I think it might also be useful to have a link from the execution_mode/async section of the configuration options ( https://github.com/bensheldon/good_job#configuration-options ) to the full async docs ( https://github.com/bensheldon/good_job#execute-jobs-async--in-process )

Ideally I would have liked all the async docs to be in one place - I feel like I had to jump around the README to find everything which was a bit confusing.

When I was trying to figure this out I definitely searched the README and issues for anything with "async" to make sure I wasn't missing anything. I also searched for "job(s) not running" and "job(s) not starting". Once I had more of clue about what was going on I started searching for things like "Job not starting when app boots" and "ActiveSupport.on_load(:active_record) async".

That's about all I can remember - hope that helps :)

[bensheldon]

@isaac Just wanted to thank you again for opening this issue. I think I finally came around to the necessity of addressing this, and there should be a fix in the latest release: https://github.com/bensheldon/good_job/releases/tag/v3.27.1