arnoldasgudas / Hangfire.MySqlStorage

MySql storage for Hangfire - fire-and-forget, delayed and recurring tasks runner
GNU Lesser General Public License v3.0
174 stars 115 forks source link

DeadLock #63

Open yuzd opened 5 years ago

yuzd commented 5 years ago

https://github.com/arnoldasgudas/Hangfire.MySqlStorage/issues/18

in the lastest version , i still get deadlock err:

LATEST DETECTED DEADLOCK

2019-06-12 07:30:16 0x7f95ffb6d700 (1) TRANSACTION: TRANSACTION 31420005, ACTIVE 0 sec starting index read mysql tables in use 1, locked 1 LOCK WAIT 7 lock struct(s), heap size 1136, 4 row lock(s) MySQL thread id 15542, OS thread handle 140282834777856, query id 57808286 172.17.0.2 root Searching rows for update update hangfire_JobQueue set FetchedAt = UTC_TIMESTAMP(), FetchToken = '97db5d2e-d886-46cc-af80-54fb649c9054' where (FetchedAt is null or FetchedAt < DATE_ADD(UTC_TIMESTAMP(), INTERVAL -1800 SECOND)) and Queue in ('default','apis','recurring') LIMIT 1 (1) WAITING FOR THIS LOCK TO BE GRANTED: RECORD LOCKS space id 33 page no 3 n bits 88 index PRIMARY of table hangfire.hangfire_JobQueue trx id 31420005 lock_mode X locks rec but not gap waiting Record lock, heap no 15 PHYSICAL RECORD: n_fields 7; compact format; info bits 0 0: len 4; hex 8000148c; asc ;; 1: len 6; hex 000001df6e38; asc n8;; 2: len 7; hex 32000001811450; asc 2 P;; 3: len 4; hex 80003985; asc 9 ;; 4: len 9; hex 726563757272696e67; asc recurring;; 5: len 5; hex 99a3587790; asc Xw ;; 6: len 30; hex 63633731373138342d626161632d346139322d613339322d316437663532; asc cc717184-baac-4a92-a392-1d7f52; (total 36 bytes);

(2) TRANSACTION: TRANSACTION 31419960, ACTIVE 0 sec updating or deleting mysql tables in use 1, locked 1 4 lock struct(s), heap size 1136, 8 row lock(s), undo log entries 1 MySQL thread id 15485, OS thread handle 140282217027328, query id 57808157 172.17.0.2 root updating update hangfire_JobQueue set FetchedAt = UTC_TIMESTAMP(), FetchToken = 'cc717184-baac-4a92-a392-1d7f52b5db74' where (FetchedAt is null or FetchedAt < DATE_ADD(UTC_TIMESTAMP(), INTERVAL -1800 SECOND)) and Queue in ('default','apis','recurring') LIMIT 1 (2) HOLDS THE LOCK(S): RECORD LOCKS space id 33 page no 3 n bits 80 index PRIMARY of table hangfire.hangfire_JobQueue trx id 31419960 lock_mode X Record lock, heap no 9 PHYSICAL RECORD: n_fields 7; compact format; info bits 0 0: len 4; hex 80001486; asc ;; 1: len 6; hex 000001df6e32; asc n2;; 2: len 7; hex 2c0000027c3024; asc , |0$;; 3: len 4; hex 8000397f; asc 9 ;; 4: len 9; hex 726563757272696e67; asc recurring;; 5: len 5; hex 99a3587790; asc Xw ;; 6: len 30; hex 61333636326665362d663163312d343133392d396639652d343261653036; asc a3662fe6-f1c1-4139-9f9e-42ae06; (total 36 bytes);

image

do you have any ideas about deadlock?

tvilkov commented 3 years ago

Hi, I also faced some deadlocks and noticed a problem in MySqlJobQueue.cs (line 42):

// Istead of 
using (new MySqlDistributedLock(_storage, "JobQueue", TimeSpan.FromSeconds(30), _options))
// it should be
using (new MySqlDistributedLock(_storage, "JobQueue", TimeSpan.FromSeconds(30), _options).Acquire())
// Otherwise distributed lock is not actually taken and this increases contention on JobQueue table