Error: "no such table: solid_queue_recurring_tasks"

[jherdman]

Greetings!

I'm all fired up after Rails World, and I'm getting started with a new Rails 8 application. I've installed this gem, along with mission_control-jobs. My stack is otherwise just a brand new Rails 8 application using SQLite.

I'm seeing the following error when trying to access the "Recurring" tab for Mission Control:

SQLite3::SQLException: no such table: solid_queue_recurring_tasks 

I can see the table listed in db/queue_schema.rb, but it's not listed in the tables of my development database either. I'm not really sure what's going on beyond these simple facts.

[rosa]

Hey @jherdman, thanks for trying this! Rails World was epic 🌍 💎 🙇‍♀️

Just to be sure, did you run bin/rails db:prepare after solid_queue:install? What version of Solid Queue are you using?

[jherdman]

Hi Rosa.

Yup! More or less...

1. rails new my_app
2. rails solid_queue:install
3. Ensure SQ is set up as my queue adapter
4. rails db:create db:migrate
5. bin/jobs 💥 fails with the error noted above 💥

It's super weird: all of the other tables are listed except recurring tasks.

[rosa]

That's super odd indeed, especially in a new app. I was going to ask whether you were using Solid Queue before, because if you were, that could certainly be the problem, but this is a new Rails app. Could you try resetting the DB? With bin/rails db:reset.

[jherdman]

No luck :(

[rosa]

What version of Solid Queue are you using?

[jherdman]

1.0

[rosa]

I've tried to reproduce following the same steps with a new app using Rails 8.0.0.beta1 but I can't, all the tables get created just fine as expected. First, are you running this in the development environment? At first, I thought it was production but seeing what you run, it seems it's development. Solid Queue is not configured by default in development so that could be the issue. Could you copy the contents of your development configuration in database.yml? Are you also setting

config.solid_queue.connects_to = { database: { writing: :queue } }

in config/environments/development.rb?

[jherdman]

Yes, definitely a development environment. I'm using devcontainers if that's at all relevant?

I copied over the configuration you mentioned to my config/environments/development.rb file and recreated my DB. No luck. Here's the relevant files you've asked for:

# config/environments/development.rb
# frozen_string_literal: true

require 'active_support/core_ext/integer/time'

Rails.application.configure do
  # Settings specified here will take precedence over those in config/application.rb.

  # Make code changes take effect immediately without server restart.
  config.enable_reloading = true

  # Do not eager load code on boot.
  config.eager_load = false

  # Show full error reports.
  config.consider_all_requests_local = true

  # Enable server timing.
  config.server_timing = true

  # Enable/disable Action Controller caching. By default Action Controller caching is disabled.
  # Run rails dev:cache to toggle Action Controller caching.
  if Rails.root.join('tmp/caching-dev.txt').exist?
    config.action_controller.perform_caching = true
    config.action_controller.enable_fragment_cache_logging = true
    config.public_file_server.headers = { 'cache-control' => "public, max-age=#{2.days.to_i}" }
  else
    config.action_controller.perform_caching = false
  end

  # Change to :null_store to avoid any caching.
  config.cache_store = :memory_store

  # Store uploaded files on the local file system (see config/storage.yml for options).
  config.active_storage.service = :local

  # Don't care if the mailer can't send.
  config.action_mailer.raise_delivery_errors = false

  # Make template changes take effect immediately.
  config.action_mailer.perform_caching = false

  # Set localhost to be used by links generated in mailer templates.
  config.action_mailer.default_url_options = { host: 'localhost', port: 3000 }

  # Print deprecation notices to the Rails logger.
  config.active_support.deprecation = :log

  # Raise an error on page load if there are pending migrations.
  config.active_record.migration_error = :page_load

  # Highlight code that triggered database queries in logs.
  config.active_record.verbose_query_logs = true

  # Append comments with runtime information tags to SQL queries in logs.
  config.active_record.query_log_tags_enabled = true

  # Highlight code that enqueued background job in logs.
  config.active_job.verbose_enqueue_logs = true
  config.active_job.queue_adapter = :solid_queue
  config.solid_queue.connects_to = { database: { writing: :queue } }

  # Raises error for missing translations.
  # config.i18n.raise_on_missing_translations = true

  # Annotate rendered view with file names.
  config.action_view.annotate_rendered_view_with_filenames = true

  # Uncomment if you wish to allow Action Cable access from any origin.
  # config.action_cable.disable_request_forgery_protection = true

  # Raise error when a before_action's only/except options reference missing actions.
  config.action_controller.raise_on_missing_callback_actions = true

  # Apply autocorrection by RuboCop to files generated by `bin/rails generate`.
  config.generators.apply_rubocop_autocorrect_after_generate!

  ## LOOKBOOK

  config.view_component.default_preview_layout = 'component_preview'

  config.lookbook.preview_display_options = {
    theme: %w[light dark] # dynamic 'theme' display option
  }
end

# database.yml

# SQLite. Versions 3.8.0 and up are supported.
#   gem install sqlite3
#
#   Ensure the SQLite 3 gem is defined in your Gemfile
#   gem "sqlite3"
#
default: &default
  adapter: sqlite3
  pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
  timeout: 5000

development:
  <<: *default
  database: storage/development.sqlite3

# Warning: The database defined as "test" will be erased and
# re-generated from your development database when you run "rake".
# Do not set this db to the same as development or production.
test:
  <<: *default
  database: storage/test.sqlite3

# Store production database in the storage/ directory, which by default
# is mounted as a persistent Docker volume in config/deploy.yml.
production:
  primary:
    <<: *default
    database: storage/production.sqlite3
  cache:
    <<: *default
    database: storage/production_cache.sqlite3
    migrations_paths: db/cache_migrate
  queue:
    <<: *default
    database: storage/production_queue.sqlite3
    migrations_paths: db/queue_migrate
  cable:
    <<: *default
    database: storage/production_cable.sqlite3
    migrations_paths: db/cable_migrate

[rosa]

Ahhh! The issue is that the DB configuration is missing in development, here:

# database.yml

# ...

development:
  <<: *default
  database: storage/development.sqlite3

Could you change this to be:

development:
  primary:
    <<: *default
    database: storage/development.sqlite3
  queue:
    <<: *default
    database: storage/development_queue.sqlite3
    migrations_paths: db/queue_migrate

[jherdman]

Hmmm... I think we may have bumped into a Rails bug:

vscode ➜ /workspaces/my_app (main) $ rails db:prepare

bin/rails aborted!
ArgumentError: No database file specified. Missing argument: database (ArgumentError)

          raise ArgumentError, "No database file specified. Missing argument: database"
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Tasks: TOP => db:prepare
(See full trace by running task with --trace)

[rosa]

Oh, that could be wrong indenting in the yaml file if you copied what I wrote above, as I wrote it directly in GitHub's comment box, so the indentation might be off 😅

[jherdman]

Totally a copy/paste error on my part!

Pasting to help any future travellers:

development:
  <<: *default # <<< THIS IS MY MISTAKE! DELETE THIS LINE
  primary:
    <<: *default
    database: storage/development.sqlite3
  queue:
    <<: *default
    database: storage/development_queue.sqlite3
    migrations_paths: db/queue_migrate

---

OK, so the big take away (and surprise) for me was that SQ wasn't configured for development out of the box. This is a bit surprising as other queue systems (e.g. Good Job, Sidekiq) work in development out of the box.

If you're amenable we could solve for the surprise with documentation?

[rosa]

Got it!

OK, so the big take away (and surprise) for me was that SQ wasn't configured for development out of the box. This is a bit surprising as other queue systems (e.g. Good Job, Sidekiq) work in development out of the box.

Rails doesn't configure solid_queue in development because it's intended to be used in production, and the async adapter in development. There's a small note about this in the Installation section in the README, but very small 😅 Improving this is already recorded in https://github.com/rails/solid_queue/issues/332, I think that would cover this.

[jherdman]

Yeah, agreed re #332 . I'm going to close this as resolved.

Many thanks for the assist!

[gczh]

That's super odd indeed, especially in a new app. I was going to ask whether you were using Solid Queue before, because if you were, that could certainly be the problem, but this is a new Rails app. Could you try resetting the DB? With bin/rails db:reset.

Hey rosa! I'm encountering a similar issue. db/queue_schema.rb is empty as well. Although, I can see that rails db:reset does create the queue database. But running rail db:migrate essentially does nothing for queue schema.

[rosa]

But running rail db:migrate essentially does nothing for queue schema.

Yes, that's expected! db:migrate would run pending migrations, but there would be none for the queue DB.

[gczh]

But running rail db:migrate essentially does nothing for queue schema.

Yes, that's expected! db:migrate would run pending migrations, but there would be none for the queue DB.

Makes sense! Strange thing though is that rails db:prepare does nothing either.

[rosa]

Huh, that should work, db:prepare should create your queue DB if it doesn't exist, and load the schema. Maybe this is because your DB already exists and is loaded, or is it because your db/queue_schema.rb file is empty?

[gczh]

Huh, that should work, db:prepare should create your queue DB if it doesn't exist, and load the schema. Maybe this is because your DB already exists and is loaded, or is it because your db/queue_schema.rb file is empty?

Ahh, you're right. Somehow, db/queue_schema.rb gets wiped sometimes.

[rosa]

@gczh, that's a Rails bug that was already fixed: https://github.com/rails/rails/issues/52829

[mediafinger]

Rails doesn't configure solid_queue in development because it's intended to be used in production, and the async adapter in development.

Personally I think this is a really odd choice. As soon as your Rails app uses any database adapter, using solid_queue in development comes basically for free. And I would bet that every development team strifes to use a setup as similar to production in development, so I really doubt the async adapter is actually used in many projects.

Maybe it makes the initial setup a bit simpler. But it seems to have the potential to cause a bit of confusion shortly after.

[uurcank]

@rosa +1 @mediafinger, solid_queue might just be enabled in development as well and if anyone wants to test things async they can just use perform_now.