Allow priority to be globally disabled/ignored

[bensheldon]

It would be nice if GoodJob could be configured to disable/ignore priority as a job feature. e.g. config.good_job.enable_active_job_priority = false. Why?

• For performance reasons: it's not possible to create an efficient database index for time-based (e.g. scheduled and FIFO) and ordered by priority. That requires either:
  • Using an index ordered on priority, created_at and then doing a condition check on scheduled_at (this is what I find most query planners choose)
  • Using an index ordered on scheduled_at that then does a reorder on priority
• I believe priority only makes sense when either queues are not well-designed, or there is insufficient capacity. Either way, there is a better action to take than introducing priority (redesigning queues based on SLOs; adding capacity) when the primary success metric is reducing queuing latency (as I believe it should be)
• GoodJob provides ample tools for executing different named queues (separately, concurrently, combined).

I do observe though that developers like to name queues based on the constrained resource (e.g. email_backend) and then use priority to hint at the SLO. I imagine people will want to preserve that organization style, so I'll probably need to introduce something like job "labels" or "tags" (which maybe connects to #753)

[Wittiest]

Hey @bensheldon,

We're looking to boost our performance with good_job and I think this is the lowest effort change I've seen in the discussions/issues on the repository.

I noticed this message from you in another comment:

I think the most delicate part of providing different job querying patterns is database indexing: GoodJob would need to have migrations that create indexes for both patterns (over-indexing is its own performance problem), or describe that it's not as simple as only setting a configuration option. I'm approaching the "Maybe I should write a book about GoodJob"-stage 😁

Would you be able to let me know your latest thoughts around the best implementation path? With some high-level guidance, I'd be happy to take this on