collectiveidea / delayed_job

Database based asynchronous priority queue system -- Extracted from Shopify
http://groups.google.com/group/delayed_job
MIT License
4.81k stars 955 forks source link

Delayed Job issue with Mongo transactions and job reservation #1144

Closed TheJohnzo closed 3 years ago

TheJohnzo commented 3 years ago

Hello, I'm running into some Mongo transaction issues. I have a cron task that queues up hundreds of delayed job records to be consumed by 2 workers. Each of those jobs may also trigger some other delayed jobs to be consumed by a different worker. (We are running 6 workers, total). The issue comes in below (see log) and crashes the worker process:

Log entry:

I, [2021-05-26T14:21:57.379990 #835]  INFO -- : 2021-05-26T14:21:57-0700: [Worker(delayed_job.0 host:jobsserver.staging.webserver.com pid:835)] 

Error while reserving job: Retryable write with txnNumber 11 is prohibited on session eae817b0-643f-4a5a-aacc-baae7ff9dbd9 - S8oMyTnRmLmiiib96wHuUHEe1v5gZMNm0MYQT5hrBs4= because a newer retryable write with txnNumber 406 has already started on this session. (225) (on web-staging-shard-00-00.stlfg.mongodb.net:27017, modern retry, attempt 1)

Could these be a bug?

Other system details: ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux]

    rails (5.2.5)
    mongoid (7.0.5)
      activemodel (>= 5.1, < 6.1)
      mongo (>= 2.5.1, < 3.0.0)
    delayed_job (4.1.9)
      activesupport (>= 3.0, < 6.2)
    delayed_job_mongoid (2.3.1)
      delayed_job (>= 3.0, < 5)
      mongoid (>= 3.0, < 8)
      mongoid-compatibility (>= 0.4.0)
albus522 commented 3 years ago

You are looking for https://github.com/collectiveidea/delayed_job_mongoid