Lock became infinite?

[marcinn]

Hi.

I have a lock defined as:

    lock = redis.Lock(
        "servicebook.tasks.emit_servicebook_notifications",
        expire=240,
        auto_renewal=True
    )

I'm acquiring it that way:

if lock.acquire(blocking=False):
  try:
    ...
  finally:
    lock.release()

Today our customer reported us an issue. Our background task, which relies on redis lock, was not executing. I've checked the key and I've found that:

127.0.0.1:6379> GET "lock:servicebook.tasks.emit_servicebook_notifications"
"GxwErhZvp2QmLvacH/pVgjGj"
127.0.0.1:6379> TTL "lock:servicebook.tasks.emit_servicebook_notifications"
(integer) -1

The TTL was set to infinite, so lock was always held.

Before I switched to lock.acquire(blocking=False) we had a code ran with blocking=True. Our background tasks are managed by RQ and sometimes they were killed due to timeout. To avoid timeouts I changed blocking=True to blocking=False. There were no other changes, upgrades nor manual changes to Redis db.

I wonder how it is possible that lock automatically changed its expiration time to infinite. There is a risk that this would happen again. Any thoughts? Feel free to ask anything If I can help somehow.

NOTE: We have several machines running same app servers (failover & load balancing).

[ionelmc]

Just to make sure I understood your scenario correctly - you always define a lock with an expire option yes?

[marcinn]

Yes.

[marcinn]

I forgot to mention that we're using redis_lock 3.5.0.

[ionelmc]

Ok so I did some testing locally. It turns out one could do "lock.expire(-1)". I think negative values should be disallowed. I'll implement a fix.

[ionelmc]

Just released v3.6.0. I hope it may fix the problem, or help you with the more strict option validation.

[marcinn]

Many thanks! I'd upgrade ASAP.