Question: How do I start specific workers only?

[harry-herskowitz]

I am trying to split queues into different workers that I can turn on and off easily, something like this:

Procfile:

worker_couchbase: QUEUE=couchbase bundle exec rake solid_queue:start
worker_scheduler: QUEUE=scheduler bundle exec rake solid_queue:start

Config File:

 workers:
    - queues: 'couchbase'
      threads: 3
      processes: 1
      polling_interval: 0.1
    - queues: 'scheduler'
      threads: 3
      processes: 1
      polling_interval: 0.1

What's the proper syntax for starting just one specific worker?

[rosa]

Hey @harry-herskowitz! With the current configuration, just running solid_queue:start, the workers will start each one in its own process, so you can terminate them individually.

If you want to have two different supervisors, each one supervising a single worker, then you'd need to have two config files, one for each worker, and set the config file via SOLID_QUEUE_CONFIG. I'm planning to build a proper CLI and add this as an option but in the meantime that'd be the way. Something like:

SOLID_QUEUE_CONFIG=config/solid_queue/couchbase.yml bundle exec rake solid_queue:start

And then config/solid_queue/couchbase.yml would look like:

workers:
   - queues: 'couchbase'
     threads: 3
     processes: 1
     polling_interval: 0.1

[harry-herskowitz]

So it seems to be ignoring my config and running all queues anyway: SolidQueue-0.3.3 Started Worker (37.6ms) pid: 11, hostname: "e1abb1c8-758d-4c8c-b5d7-3227972dd01f", polling_interval: 0.1, queues: "*", thread_pool_size: 3 I'm just trying to get it to work with the default config file:

default: &default
  dispatchers:
    - polling_interval: 1
      batch_size: 500
      concurrency_maintenance_interval: 300
  workers:
    - queues: 'scheduler'
      threads: 3
      processes: 1
      polling_interval: 0.1

development:
  <<: *default

test:
  <<: *default

production:
  <<: *default

[rosa]

Huh, that's quite strange! It seems as if it was using the default configuration because it can't find yours. Where is your configuration file located?

[harry-herskowitz]

config/solid_queue.yml

[rosa]

Hmm... so strange. It should work and I can't reproduce this. I copied your configuration and started solid queue and it correctly starts a single worker for the "scheduler" queue 😕

Is it possible the config/solid_queue.yml file is stale in the place where you're starting Solid Queue? Are you setting SOLID_QUEUE_CONFIG to another value?

[harry-herskowitz]

hmm, so I have 2 "review" apps that share a db and run solid queue. I don't see how the configs would cross though, unless that gets loaded into the db?

[rosa]

No, the configuration doesn't get loaded into the DB. However, running two different apps with the same Solid Queue DB might cause some problems with jobs from one app running in the other. In any case, not related to the problem here, which is that Solid Queue seems to be ignoring your config file 😕 Have you checked that the file is the right one for the app you're testing this with, and that SOLID_QUEUE_CONFIG is not pointing to a different config?

[harry-herskowitz]

Okay, so I ran the installer fresh to get the original config file and configured it from scratch. Now it's working. Not sure why since the file was identical but maybe Heroku just wasn't picking up the changes? Anyway, thanks for your help!

[rosa]

Ohh, glad to hear! So strange it didn't work before 😕

Going to close this one but let me know if you're still running into trouble 🙏

[harry-herskowitz]

Oh I see I think the indentation may have been off