Per lock timeout values do not appear to work

[jdejong]

I have the following lock defined in a class

lock :service, :expiration => 15.minutes, :timeout => 20.seconds 

I have also tried several other variations

lock :service, :expiration => 15.minutes, :timeout => 20 
lock :service, { :expiration => 15.minutes, :timeout => 20.seconds }

for setting the lock up. However when I use the lock and I cause a collision that should block the second thread for 10-15 seconds I end up with the following error.

Timeout on lock service:1:service_lock exceeded 5 sec

It seems like it is not taking the options I am passing into the lock and just using the default.

Not that I think it matters but this is how I am grabbing the lock

service.service_lock.lock do

I am on Ruby 1.9.3p448, redis 3.0.7 and redis-objects 0.9.1.

Thanks in advance for any suggestions.

[nateware]

I'm not sure what to do here, as there are tests for this in the gem (I think). Any chance you have timeout code somewhere else that could be causing this?

[jdejong]

Not that I know of. The only timeout besides the lock is on the connection pool for redis which is 5 seconds (very similar setup to your example in the readme). I also played with different values and it seemed to stick with 5 seconds. I have code that is running in Rails Controllers, Resque Workers and Ruby Daemons and I have seen this error from the Workers and the Daemons but I don't recall seeing it in the controllers if that helps any. But I don't see how that would change anything.

[nateware]

Can you please try redis-objects 1.2.0 which includes a new internal implementation of expiration? References #173 and #174

[jdejong]

After a good amount of testing in production and staging we are still seeing

Timeout on lock service:2296:service_lock exceeded 5 sec 

where service lock is defined as

lock :service, :expiration => 15.minutes, :timeout => 20.seconds 

So it would appear this issue is still occurring.

[jdejong]

On top of that we added a new lock to our code and often the lock is getting left behind in Redis. We are stuck on a slightly older version of Redis gem due to some dependencies (3.0.7) and we use RedisLabs MultiAZ cluster. Any thoughts?

[nateware]

I'm definitely interested in root causing this and fixing it, but my time the next few weeks is extremely short. If somebody could at least create a test case that reliably fails, that would be a big step in fixing it.

On Sat, Jul 11, 2015 at 4:21 PM, jdejong notifications@github.com wrote:

On top of that we added a new lock to our code and often the lock is getting left behind in Redis. We are stuck on a slightly older version of Redis gem due to some dependencies (3.0.7) and we use RedisLabs MultiAZ cluster. Any thoughts?

— Reply to this email directly or view it on GitHub https://github.com/nateware/redis-objects/issues/163#issuecomment-120668944 .

[mooncowboy]

@jdejong having the same problem as you, also using RedisLabs. Stuck on redis 3.2.2 and redis-objects 0.7.0 gems because of dependencies. @nateware working on getting a test case but can't make it work. Only fails in production and only sometimes. The lock gets left behind.

[byronalfonso]

@theplastictoy were you able to resolve this issue?

[mooncowboy]

@byronalfonso no. We're migrating to a newer stack version, so in the meantime we're doing it differently. I would say this is a coincidence of old versions of redis-objects and rails.

[byronalfonso]

@theplastictoy I see.. Sounds like a good route to go to.. Thanks for the reply 👍
Btw, it turned out that my problem wasn't really the timeout but that the services are needed be unlocked using RDM.. It's working well on my end now.. thanks and credits to my seniors who helped me figure this out :)

[jdejong]

I think I might have finally figured out the issue. In cases where the class locks are using used, it doesn't seem like the options are reaching the Lock class from the Object:Lock class. The reason I think this is the issue is any class lock always sets the key's value to 1 but when I just the Lock class directly it sets the timestamp as expected. Do the service locks just not work?

[nateware]

Can you see if redis-objects v1.4.3 just released to rubygems improves locking, for those of you still interested?

[nateware]

CC @tmsrjs to potentially verify the tests are actually working