samcook / RedLock.net

An implementation of the Redlock algorithm in C#
MIT License
947 stars 167 forks source link

Dead lock on extend fail #93

Closed Cugap closed 2 years ago

Cugap commented 2 years ago

enviroment: 1 noded redis 5 nodes of app

algo: after application start they try to get lock(with params .CreateLockAsync(key, expiryTime: 2 minutes, waitTime: 1minute, retryTime: 30 seconds)) and if this operation ends with success start process distributed queue, also in that time started timer that check lock status every 2 minutes and if it is not aquired then stop processing and try to lock again.

problem: sometimes, quite often after restart, firstly lock gets one instance of app, but after 2 minutes another app gets it too, its not a problem for algo, but first instance still think that isAquired == true, so two instance process data in same time.

problem 2: today after restart lock gets only one app so im delete lock from redis for restart immitation and catch more strange behavior - app that have lock before removal notice that lock lost and try to relock, app 2 gets lock, but then every 2 minutes another app get lock, and after all apps locked they stops to extend, all have 0 extends, and there is no lock key in redis

timelapse, read from down

app5: 2022-02-03 09:48:21.1104|DEBUG|Lock IsAcquired with id 731334ee-2548-4ff3-8f9c-a3036abd12ae

app4: 2022-02-03 09:46:18.7026|DEBUG|Lock IsAcquired with id 8921325f-2dfc-4f18-921a-8aaf0c662eec

app3: 2022-02-03 09:44:09.1532|DEBUG|Lock IsAcquired with id 0405daf7-cf8e-42f4-8838-e03ba5234af6

app1: 2022-02-03 09:41:47.5693|DEBUG|Lock IsAcquired with id 129741b1-f85e-4aa6-9def-95b7d30a1831

app1: 2022-02-03 09:41:17.2084|DEBUG|Lock lost with id d954b8e8-dbec-44a7-80ff-695897faf4b5

app2: 2022-02-03 09:39:45.7486|DEBUG|Lock IsAcquired with id 5368b2ec-a456-49fe-80f8-1894f39610dd

somewhere here lock key deleted from redis

app1: 2022-02-03 09:19:17.1875|DEBUG|Lock IsAcquired with id d954b8e8-dbec-44a7-80ff-695897faf4b5

samcook commented 2 years ago

Hi, this sounds like it could be the issue raised in #97.

Please try updating to RedLock 2.3.2 and let me know if you still experience the problem.

Cugap commented 2 years ago

Hello, yep, that helps in my case, thx!