search

ionelmc / python-redis-lock

Lock context manager implemented via redis SET NX EX and BLPOP.
https://pypi.python.org/pypi/python-redis-lock
BSD 2-Clause "Simplified" License
549 stars 77 forks source link

More granular timeouts #16

Open Suor opened 9 years ago

Suor commented 9 years ago

Redis supports millisecond timeouts on SET with PX, but BLPOP understands only integer seconds. This is still achievable by using publish/subscribe with a bit more hustle.

ionelmc commented 9 years ago

Do you need this feature for cacheops?

Suor commented 9 years ago

Yes. I plan to expose this timeout as user setting. It also makes sense to set it to same time as calculation will take, which is usually under second in web apps.

ionelmc commented 9 years ago

Just to understand this correctly, I wonder - wouldn't that create a smaller dogpile effect if the database and management overhead (query serialization, deserialization and cache storage) takes more than 1 second?

If we use pubsub then all the clients would be woken up, even if they can't get the lock.

On the other hand, this slightly less efficient technique could be active only if the timeout is a fraction.

Suor commented 9 years ago

Look if I set expire to 0.5 second then I am still sure that calculation is made no more than twice a second, which should be enough to mitigate dogplie.

OTOH, in normal operation things are unlocked as soon as possible, so no need to make timeout that smaller. So for cacheops this should be ok current way.

ionelmc commented 9 years ago

Now I'm not sure if you still want this feature. Should I take this as a "not for now"?

Suor commented 9 years ago

I don't, at least for cacheops. Probably, you do? )

jhorman commented 9 years ago

Sorry to hijack a thread for a question, but here goes. From what I understand, when Redis runs low on memory, it will randomly sample keys, and evict the ones with the shortest remaining ttl. That means that you cannot rely on your expire ttl to be accurate when churning the cache. That means under memory constraints, locks might disappear if using expires. It might be safer to not use redis expires, and maintain your own expirations.

Or, I have this all wrong.

zoni commented 9 years ago

@jhorman (and anyone else coming by this issue), what Redis does when running out of memory depends on the specific eviction policy you have configured.

For this use-case (distributed locking, especially with low-TTL keys), setting a noeviction policy might be wise.

dimaqq commented 8 years ago

Please document noeviction requirement.