"Clean up" job queues

[sshetenhelm]

Story We should examine and "clean up" our job queues for our various jobs. This may include investigating the following:

• What are the priority weights for all jobs, and do those weights make sense in the current job landscape?
• Can we limit workers to accepting only up to "X" number of certain job types, or a total of "X" weight for all jobs on a worker?

Acceptance

• [x] Identify and document priority weights for all jobs
• [x] Decide if weight changes should be made and, if so, make changes
• [x] Investigate if workers can be limited to only "X" number of certain job types
• [x] Investigate if workers can be limited to a total weight of "X" for all jobs on a worker
• [x] Make necessary changes or, if work is complicated, create new tickets for individual pieces of work

[jpengst]

Here's a list of all our jobs ordered by priority. It looks like we can limit the number of jobs that get enqueued by a single job by setting a job_limit (kind of similar to the SolrIndexAllJob) and we can also limit the total number of jobs that can be enqueued through a GoodJobs config: config.good_job.queue_select_limit = 1000

I think we should discuss the priorities listed below to see if we need any adjusting:

[sshetenhelm]

FOR YOUR CONSIDERATION, comments and dissenions welcome.

To be lowered in priority:

• GeneratePDFJob - Reduce from 50 to 10, puts on par with GeneratePtiffJob

To be raised in priority:

The following should be raised to 10 -

• SyncFromPreservicaJob

In theory, this would allow Preservica Syncs to be slightly more prioritized than, say, resyncing an entire collection's metadata.

The following should all be raised to 50 -

• ExportParentMetadataCsvJob
• ReassociateChildOidsJob
• GenerateOutputCsvJob
• ExportAllParentSourcesCsvJob
• ProblemReportManualJob

Basically, with limited exceptions, any job that is user generated via a batch process or just results in a spreadsheet should be prioritized.

• UpdatePermissionRequestsJob - Is this the one that checks to make sure people have or do not have permissions to view on object? If that is correct, we'd like this bumped up to 100, as in my opinion, it should be one of the more important jobs.

[jpengst]

"Is this the one that checks to make sure people have or do not have permissions to view on object?"

^ Yep, that is correct.

PR for the adjustments: https://github.com/yalelibrary/yul-dc-management/pull/1433

[K8Sewell]

Deployed to Test with release v2.71.9

[K8Sewell]

On Test, a sample of jobs are completing successfully - except Preservica Resync jobs which is to be expected. Not really possible to test the changed priorities effect on how quickly jobs are processed without a significant amount of jobs running. The ones I tested went really quickly. Screenshots below of updated priority reflected in GoodJob Dashboard and successfully completed jobs.

Will promote to UAT.

[sshetenhelm]

Could someone please remind me which job exports all parent objects by set? I triggered that one as well as a 'by source' and the 'by source' worked, but I'm assuming the 'by set' is still in the queue after a bunch of manifest refreshes.

The 'export all parents by admin set' should also be set at 50.

[jpengst]

The 'export all parents by admin set' is handled by the CreateParentOidCsvJob and is currently set to -100. Will put back to In Progress for the quick change.

PR: https://github.com/yalelibrary/yul-dc-management/pull/1441

[K8Sewell]

Deployed to Test with release v2.72.1

[K8Sewell]

On Test 'export all parents by admin set' job is set to 50 as expected. Will promote to UAT.

[sshetenhelm]

Looks good to me! Ready to close