Sidekiq jobs are getting run twice, leading to database locks

[maxkadel]

For more detailed notes, see the datadog notebook

Ensure Sidekiq jobs are not created twice with the same job id. This is very likely a redis latency issue.

History

We have a monitoring alert that is going off repeatedly for bibdata, saying that postgres queries are taking a very long time - sometimes as much as 15 minutes (at which point they're probably getting killed by Postgres, not finishing).

The postgres queries that are taking so long are:

SELECT dump_types . * FROM dump_types WHERE dump_types . constant = ? LIMIT ?

Which seem to be getting called from the AlmaDumpTransferJob.

It seems possible that this job is somehow getting called twice with different GlobalIDs, and causing a database lock? If that's it, it could be a redis latency issue? Or a sidekiq thread management issue? (again, see datadog notebook)

Acceptance Criteria

• [x] Reconfigure staging to use central staging redis server
• [x] Reconfigure qa to use central staging redis server
• [ ] Reconfigure production to use central production redis server. (We need to make sure to flush existing jobs and stop workers prior to switching to this in production).
• [ ] Check journalctl -u bibdata-workers.service --grep AlmaDumpTransferJob on the worker box to confirm this issue isn't happening any longer.

[maxkadel]

Sidekiq thread on migrating to a new Redis suggests using redis replication

To migrate, you set up a new replica of the old primary, let it replicate, shut down workers, shut down primary, promote new replica to new primary, start up workers.

[Beck-Davis]

• [x] change BIBDATA_REDIS_URL in group_vars
• [x] ssh to each worker box & make sure no jobs are running/queued
• [x] stop sidekiq workers
• [x] run entire bibdata playbook from branch ~with site-config flag~
• [x] restart sidekiq workers on the boxes
• [x] make sure sidekiq is still running
• [x] run a background job on bibdata & check how many jobs are run

[hackartisan]

Please make sure to use the central redis var described in https://github.com/pulibrary/princeton_ansible/blob/main/roles/redis/README.md

[maxkadel]

Thanks @hackartisan! I had missed this.

[maxkadel]

Is Sidekiq connecting to Redis twice?

NFO: Sidekiq Pro 7.2.0, commercially licensed. Thanks for your support!
Apr 08 06:27:53 bibdata-alma-worker-staging1 sidekiq[865]: 2024-04-08T10:27:53.704Z pid=865 tid=49l INFO: Sidekiq 7.2.2 connecting to Redis with options {:size=>10, :pool_name=>"internal", :url=>"redis://lib-redis-staging1.princeton.ed>
Apr 08 06:27:53 bibdata-alma-worker-staging1 sidekiq[865]: 2024-04-08T10:27:53.714Z pid=865 tid=49l INFO: Sidekiq 7.2.2 connecting to Redis with options {:size=>2, :pool_name=>"default", :url=>"redis://lib-redis-staging1.princeton.edu:>

[maxkadel]

The redis production server is still on Redis 6.0, which is too old for our current version of Sidekiq. See Princeton Ansible ticket to upgrade this server

[kevinreiss]

We should talk to ops on how to transition the production environment to the same version as staging or discuss alternative plans.

[maxkadel]

This may be related to https://github.com/pulibrary/bibdata/issues/1959. Newer Honeybadger error. May be addressed by postgres configuration change