Open hmnhf opened 2 years ago
@hmnhf thanks for opening this Issue! The Readme was recently updated to try to explain how to calculate total threads (#525) and I think it exposed a conceptual problem.
Briefly to answer your question: GOOD_JOB_MAX_THREADS
is "default threads per query execution pool"
The name predates the ability to configure multiple query pools within a single process (e.g. the --queues=mice:2;elephants:1
syntax). The value in --queues
overrides the max-threads value.
The example given for defining database.yml
's pool:
value is really just flat out wrong.
To address this:
GoodJob.max_threads
value that would be accurate at that stage of Rails initialization (it gets tricky because configuration gets loaded at different places and it might not be available at the stage that database.yml is read)GOOD_JOB_MAX_THREADS
and --max-threads
configuration options and replace them with something less absolute (e.g. GOOD_JOB_POOL_THREADS
)@bensheldon so if I have the following queue "high_priority:7;default:4;low_priority:2;*" and GOOD_JOB_MAX_THREADS = 5 that means:
7 threads for high priority 4 threads for default 2 threads for low priority 5 threads for * (This '5' comes GOOD_JOB_MAX_THREADS). Totaling 18 threads due to the queue.
Additionally, goodjob needs 1 thread for a notifier. 1 thread for a cron. 1 thread for executor. totaling 3 threads as overhead for goodjob.
With GoodJob running separately from the web process, based on the above example the database.yml should have:
pool: 21
Is that right? I spent the last couple hours reading through the readme and various issues, and that's what I have deduced....
@philipqnguyen yep, that's correct number of threads GoodJob will need from the database and the minimum value you should safely have in your database.yml
.
I'm also recommending that people don't set the minimum but rather just set like 50 or 100 and don't worry about it from the perspective of the application. You'll need to have those connections available from the database, but trying to set a minimum in the database.yml
isn't necessary and can lead to not having enough database connections available.
For example, if you're using Active Record load_async
in any jobs to further parallelize work, you'll run out of database connections in Active Records connection pool.
Thank you for confirming @bensheldon
Hi and thanks for this awesome gem!
I've become a bit confused about the
max_threads
option.From reading the following two sections, at first I thought
max_threads
is a global option that defines the maximum number of threads across all queues. (Meaning that if there are two queues named A and B, their combined total threads' count won't exceed the number ofmax_threads
.)But after reading the following two sections, I figured it's probably the default max value for each queue's threads:
But, then again in the Database Connections section, there's the following part which gives the impression that
max_threads
is the max number of threads across all different queues.And then, there's the description on how to calculate the number of required threads by GoodJob to be set as
GOOD_JOB_MAX_THREADS
.Assuming that I've understood this correctly, I think the confusion comes from the fact that the
max_threads
option and theGOOD_JOB_MAX_THREADS
used in the pool size setting can be two different values. In other words, their name could bedefault_max_threads_per_queue
andGOOD_JOB_REQUIRED_THREADS
. Have I misunderstood something or is this correct?