rosa
commented
9 months ago @andbar, could you share the LATEST DETECTED DEADLOCK section if you run SHOW ENGINE INNODB STATUS in the DB server where you have Solid Queue tables?
Open andbar opened 9 months ago
rosa
commented
9 months ago @andbar, could you share the LATEST DETECTED DEADLOCK section if you run SHOW ENGINE INNODB STATUS in the DB server where you have Solid Queue tables?
andbar
commented
9 months ago Here you go. Thanks for looking into it. Let me know if you need any more info.
rosa
commented
9 months ago @andbar, I've looked into this and I see why the deadlock is happening but it's not clear to me why the transaction (2) is locking 361 records in the index_solid_queue_dispatch_all index of solid_queue_scheduled_executions while only trying to delete 18 rows. Could you let me know:
Thank you! 🙏
andbar
commented
9 months ago Yep, here they are:
solid_queue (0.2.1) 8.0.mysql_aurora.3.03.1
dispatchers:
- polling_interval: 1
batch_size: 500Agh, @andbar, I was completely stumped because I thought the code you were running included this change https://github.com/basecamp/solid_queue/pull/155/commits/a30c2cb47a8f872c89f1643f80dec87e5f001691 that I got about 3 weeks ago (GitHub shows last week because I had rebased) and that we're running in production because we're running the branch with support for recurring tasks. I noticed that one while I was working on recurring jobs because I hit a super similar deadlock, and fixed it there. Then, when you reported this one and I looked at your SHOW ENGINE INNODB STATUS I was like "how can this be possible?!" 😆 🤦♀️ In my head, I had already shipped that fix ages ago. I think the deadlock is due to that missing job_id in the ordered scope.
I'm going to try to ship a new version with support for recurring jobs and that fix so you can test it. I'm on call this week and had a busy Monday, but hopefully will get to it tomorrow!
andbar
commented
9 months ago Ah! Haha, so easy to do. We'll be glad to test that fix with our situation. I'll watch for a new version. Thank you!
rosa
commented
9 months ago Thank you so much! I just shipped version 0.2.2 with this fix, as I didn't have time to wrap up the recurring jobs PR, so I decided to just extract that fix. Could you try this one and see if you still encounter the deadlock? 🙏 Thank you!
andbar
commented
9 months ago Hi, @rosa, we deployed that new version and thought it might have fixed it but unfortunately we got some more deadlocks today. Here's the latest deadlock log from the db, hopefully it helps pinpoint what might be causing it.
rosa
commented
8 months ago Ouch 😞 Sorry about that, and thanks for the new debug info. I'll continue looking into it.
andbar
commented
8 months ago Thanks, @rosa. I haven't had a chance to look at it much myself before this, due to some other needed work in our project that's using this gem, but I'm hoping to be able to dig into it further. I don't have much experience with deadlocks, though, so I'm trying to brush up on that first 😬.
From a first glance, it appears that maybe the issue is locking due to both transactions (the insert and the delete) needing to lock this index: index_solid_queue_dispatch_all? Or more accurately, the insert appears to lock the PRIMARY index first and then tries to acquire a lock on the index_solid_queue_dispatch_all index, while the delete is going in the opposite direction and locks the index_solid_queue_dispatch_all index first and then tries to acquire a lock on the PRIMARY index. Does that sound right?
Maybe that's why the delete transaction (transaction 2 in the logs) shows "5336 row lock(s)" even though it's only deleting something like 19 jobs - b/c of the index_solid_queue_dispatch_all index?
rosa
commented
8 months ago Hey @andbar, sorry for the delay! I haven't forgotten about this one, but I've been working on other stuff in the last few weeks. I'm back looking at this, and I wonder if https://github.com/rails/solid_queue/pull/199 might help in your case. Would you mind trying that one out if you're still using Solid Queue and experiencing this problem?
andbar
commented
7 months ago Hi @rosa, I just realized I hadn't responded to you yet, I apologize for that. We ended up moving away from Solid Queue to using something else that was just a better fit for our particular needs due to the nature of the jobs we're running, so unfortunately I won't be able to test that fix. I'm sorry!
rosa
commented
7 months ago Hi @andbar! Oh, no worries at all! Thanks so much for letting me know, really appreciate you taking the time to test and report this and your patience through the troubleshooting 🤗
paulhtrott
commented
7 months ago Hi @rosa 👋🏽
We tried your fix #199 at Pressable, it fixed our deadlock issue
Rails 7.1.2 ruby 3.2.2 MariaDB 10.6.17 (trilogy)
Thank you! 🚀
paulhtrott
commented
7 months ago Hi again @rosa. I spoke too soon. #199 did not completely get rid of our deadlock issue, but it did reduce it.
rosa
commented
6 months ago Hey @paulhtrott, ohhhh, thanks for letting me know! It's a bummer it didn't solve it completely, but I'll take the reduction. I'll merge that and will start using it in HEY, as we're going to increase the number of scheduled jobs per day in ~4M tomorrow, so hopefully we'll hit this ourselves and that will give me more info to think of other solutions.
paulhtrott
commented
6 months ago Hi @rosa we have had zero luck being able to diagnose our deadlock issue. This is how our architecture is structured:
Our errors do not show much details outside of the following, plus a stack trace (attached):
We have resorted to setting wait_untils for most of our jobs, it seems the delay helps on most occasions, but it is inconvenient in some cases.
Are there any other details that might be helpful for you?
rosa
commented
5 months ago Ohhh, @paulhtrott, that's super helpful! This looks like a different kind of deadlock than the one I tried to fix! Let me try to put together another fix for this one.
rosa
commented
5 months ago That deadlock is the same one as https://github.com/rails/solid_queue/pull/229, going to tweak that one a bit and ship.
rosa
commented
5 months ago @paulhtrott, I just released version 0.3.3 with a possible fix for this one deadlock: #240.
paulhtrott
commented
5 months ago @paulhtrott, I just released version 0.3.3 with a possible fix for this one deadlock: #240.
Thank you @rosa , we will give that a try today, I'll report back after a couple of days 🎉
paulhtrott
commented
5 months ago Hi @rosa I'm back sooner than I wanted to be 😆 . We are still having the issue after 0.3.3. Same two stack traces basically.
rosa
commented
5 months ago Ohh, bummer 😞 I'll continue working on it. Thanks a lot for letting me know!
rosa
commented
5 months ago Oh, @paulhtrott, I realised something... The query done now to delete records from solid_queue_ready_executions should be using the primary key instead of job_id, so I wonder if the locks that both transactions are waiting are still in the primary key index or another index... If you have the output from the LATEST DETECTED DEADLOCK section when you run SHOW ENGINE INNODB STATUS, would you mind sharing it?
Thank you so much again 🙏
paulhtrott
commented
5 months ago Hey @rosa! Sure, here is the output
paulhtrott
commented
2 months ago Hi @rosa! Just want to see if you have had a chance to look into the deadlock issue? Here is a new deadlock file to show that this is still happening on production.
Our Setup Linux Docker Container -> Four instances Puma version: 6.4.3 MariaDB: 10.6.17 -> Single Instance ( solid_queue tables are also here ) Ruby: 3.3.5 Rails: 7.1.3.4 Trilogy: 2.8.1 SolidQueue: 1.0.0.beta
iJackUA
commented
1 month ago I've noticed that we are receiving Deadlock error occasionally, only in places where jobs were enqueued in the loops (solid_queue 1.0.0, Rails 7.2)
smth like this
items.each do |item|
#....
ItemProcessingJob.perform_later(item, #...other params)
endIn terms of best practices, it is clearly not the best code.
Now rewriting it to utilize ActiveJob.perform_all_later, let's see would it help or not.
iJackUA
commented
1 month ago Recent refactoring of my code with ActiveJob.perform_all_later really removed Deadlocks initiated there.
But another kind of Deadlocks still present, and there are quite a lot of them.
I have diagnosed that it happens only while SolidQueue automatically adds Recurring jobs, it's quite random (does not depend on particular job), those with less interval (like each 1 munite) are seen more often than with longer interval (like each 1 hour).
The Pattern I see in Sentry tracks is always the same
SolidQueue-1.0.0 Error enqueuing recurring task (36.5ms) task: "MyJob", at: "2024-10-31T10:37:30Z"
...
SolidQueue-1.0.0 Error in thread (0.0ms) error: "ActiveRecord::Deadlocked Mysql2::Error: Deadlock found when trying to get lock; try restarting transaction"P/S DB adapter does not have a lot of difference, I've tried mysql2 and trilogy
Stacktrace part
...
"/PATH_MASK/bundle/ruby/3.3.0/gems/solid_queue-1.0.0/app/models/solid_queue/recurring_execution.rb:19:in `create_or_insert!'",
"/PATH_MASK/bundle/ruby/3.3.0/gems/solid_queue-1.0.0/app/models/solid_queue/recurring_execution.rb:29:in `block (2 levels) in record'",
"<internal:kernel>:90:in `tap'",
"/PATH_MASK/bundle/ruby/3.3.0/gems/solid_queue-1.0.0/app/models/solid_queue/recurring_execution.rb:27:in `block in record'",
"/PATH_MASK/bundle/ruby/3.3.0/gems/activerecord-7.2.1.1/lib/active_record/connection_adapters/abstract/transaction.rb:616:in `block in within_new_transaction'",
"/PATH_MASK/bundle/ruby/3.3.0/gems/activesupport-7.2.1.1/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'",
"/PATH_MASK/bundle/ruby/3.3.0/gems/activerecord-7.2.1.1/lib/active_record/connection_adapters/abstract/transaction.rb:613:in `within_new_transaction'",
"/PATH_MASK/bundle/ruby/3.3.0/gems/activerecord-7.2.1.1/lib/active_record/connection_adapters/abstract/database_statements.rb:361:in `transaction'",
"/PATH_MASK/bundle/ruby/3.3.0/gems/activerecord-7.2.1.1/lib/active_record/transactions.rb:234:in `block in transaction'",
...show engine innodb status;
------------------------
LATEST DETECTED DEADLOCK
------------------------
2024-10-31 04:57:00 0x775c42fb4640
*** (1) TRANSACTION:
TRANSACTION 16229321, ACTIVE 0 sec fetching rows
mysql tables in use 1, locked 1
LOCK WAIT 9 lock struct(s), heap size 1128, 14 row lock(s), undo log entries 6
MariaDB thread id 67254, OS thread handle 131238144460352, query id 48669755 10.0.1.160 root Updating
DELETE FROM `solid_queue_ready_executions` WHERE `solid_queue_ready_executions`.`id` IN (762906, 762907, 762911)
*** WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 588 page no 3 n bits 320 index PRIMARY of table `ijackua_queue_staging`.`solid_queue_ready_executions` trx id 16229321 lock_mode X waiting
Record lock, heap no 6 PHYSICAL RECORD: n_fields 7; compact format; info bits 0
0: len 8; hex 80000000000ba423; asc #;;
1: len 6; hex 000000f7a3c8; asc ;;
2: len 7; hex ba000001900121; asc !;;
3: len 8; hex 80000000000ba9db; asc ;;
4: len 11; hex 6865616c7468636865636b; asc healthcheck;;
5: len 4; hex 80000000; asc ;;
6: len 8; hex 99b4be4e40010b7b; asc N@ {;;
*** CONFLICTING WITH:
RECORD LOCKS space id 588 page no 3 n bits 320 index PRIMARY of table `ijackua_queue_staging`.`solid_queue_ready_executions` trx id 16229320 lock_mode X locks rec but not gap
Record lock, heap no 6 PHYSICAL RECORD: n_fields 7; compact format; info bits 0
0: len 8; hex 80000000000ba423; asc #;;
1: len 6; hex 000000f7a3c8; asc ;;
2: len 7; hex ba000001900121; asc !;;
3: len 8; hex 80000000000ba9db; asc ;;
4: len 11; hex 6865616c7468636865636b; asc healthcheck;;
5: len 4; hex 80000000; asc ;;
6: len 8; hex 99b4be4e40010b7b; asc N@ {;;
*** (2) TRANSACTION:
TRANSACTION 16229320, ACTIVE 0 sec inserting
mysql tables in use 1, locked 1
LOCK WAIT 4 lock struct(s), heap size 1128, 2 row lock(s), undo log entries 2
MariaDB thread id 63345, OS thread handle 131238143231552, query id 48669753 10.0.1.160 root Update
INSERT INTO `solid_queue_ready_executions` (`job_id`, `queue_name`, `priority`, `created_at`) VALUES (764379, 'healthcheck', 0, '2024-10-31 04:57:00.068475') RETURNING `id`
*** WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 588 page no 6 n bits 320 index index_solid_queue_poll_by_queue of table `ijackua_queue_staging`.`solid_queue_ready_executions` trx id 16229320 lock_mode X locks gap before rec insert intention waiting
Record lock, heap no 5 PHYSICAL RECORD: n_fields 4; compact format; info bits 32
0: len 8; hex 7061727469616c73; asc partials;;
1: len 4; hex 80000000; asc ;;
2: len 8; hex 80000000000ba9d2; asc ;;
3: len 8; hex 80000000000ba41a; asc ;;
*** CONFLICTING WITH:
RECORD LOCKS space id 588 page no 6 n bits 320 index index_solid_queue_poll_by_queue of table `ijackua_queue_staging`.`solid_queue_ready_executions` trx id 16229321 lock_mode X
Record lock, heap no 2 PHYSICAL RECORD: n_fields 4; compact format; info bits 32
0: len 8; hex 7061727469616c73; asc partials;;
1: len 4; hex 80000000; asc ;;
2: len 8; hex 80000000000ba9d3; asc ;;
3: len 8; hex 80000000000ba41b; asc ;;
Record lock, heap no 3 PHYSICAL RECORD: n_fields 4; compact format; info bits 32
0: len 8; hex 7061727469616c73; asc partials;;
1: len 4; hex 80000000; asc ;;
2: len 8; hex 80000000000ba9d7; asc ;;
3: len 8; hex 80000000000ba41f; asc ;;
Record lock, heap no 5 PHYSICAL RECORD: n_fields 4; compact format; info bits 32
0: len 8; hex 7061727469616c73; asc partials;;
1: len 4; hex 80000000; asc ;;
2: len 8; hex 80000000000ba9d2; asc ;;
3: len 8; hex 80000000000ba41a; asc ;;
*** WE ROLL BACK TRANSACTION (1)For those searching the workaround (which at least reduces Deadlocks for MariaDB in Galera Cluster):
Reduce transaction level isolation.
Setting transaction_isolation="READ-COMMITTED" seems to resolve deadlocks in my case.
Default level before that was REPEATABLE READ.
But it's up to you to decide is it an appropriate level of tx isolation for your app.
rosa
commented
3 weeks ago Ah, good point! If you're running Solid Queue in its own DB, READ-COMMITTED should be completely fine for Solid Queue operation.
paulhtrott
commented
1 week ago Yeah, unfortunately that doesn't work in our environment.
We are seeing some failed jobs due to hitting a deadlock when solid queue is trying to create the ScheduledExecution for the job. This is usually happening for us on jobs that are having to be retried due to a throttling constraint we are dealing with from an external api. Here is one example, with part of the backtrace. The job attempts to execute, gets the throttling constraint so it tries to schedule a retry, and it looks like it's trying to do the
ScheduledExecution.create_or_find_by!on line 40 ofapp/models/solid_queue/job/schedulable.rbwhen it hits the deadlock on the insert.Backtrace: