Releasing locks from another machine

[ghost]

Hi,

I am looking into exposing the capabilities of this library as a service. I've stumbled upon the problem of releasing the lock; since these are distributed locks, I should be able to release the lock from another machine than the one which took it.

How do I do that? I am noticing that the release logic is using IDisposable pattern which scopes them to work only on the process which took the lock in the first place.

I am trying to use Redis, but MySql is also an option for me. Thanks!

[madelson]

Hi @CatalinAdlerDF apologies for the delayed response.

The library does not natively support acquiring a lock on one machine and releasing it on another machine. Can you explain more about why you want to do that?

One option you have is to have the app that wants to trigger the release notify the app holding the lock in some way.

[ghost]

Was trying to expose the library as a service, so the request to release a lock might b served by a different instance then the one which took it.

[madelson]

@CatalinAdlerDF interesting thanks for explaining.

FWIW I feel like you'd be better off having consumers of the "locking service" just reference and use the library directly. This avoids the indirection introduced by the locking service and dodges the "release from elsewhere" issue.

Holding a lock is a fundamentally stateful operation. In some cases the state is implicit (e. g. in MySQL the state is tied to the session between the calling app and the database). In other cases, the state is explicit (e. g. in Redis we use a lockID to identify the particular acquirer).

In order to release the lock, you need to have the state. For implicit state, there's no obvious way to do this. For explicit state, in theory we could expose the lock ID and some other server could release using that same lock ID. However, even in this case you run into another problem: even for backends like Redis, there is still implicit state because the acquiring server periodically renews held locks. If the acquiring server restarts, the lock will be lost.

If you must expose locking through a server, I can see two potential paths forward:

One option is to build server affinity into your system. For example you could have a scheme which routes incoming requests to specific servers based on the name of the lock to acquire/release. You aren't going to be resilient to servers restarting anyway, so this should be feasible although if you ever want to grow the pool of servers in the future it will be a bit tricky to shuffle things around in a live environment.

Another option would be to leverage MySQL with multiplexing disabled so that you have a unique MySQL connection per acquired lock. You can then assign each connection an ApplicationName based on the lock name, and then issue a kill command based on the application name to force the release of a lock held by another server. When locks are acquired, you'd still need to store the handle in a static datastructure to prevent garbage collection, and you can leverage the HandleLostToken to perform cleanup on this datastructure. Not pretty but it should work.

In the future, we could consider an API like DangerousForceRelease() on distributed lock instances that would just release them from whoever is holding it (only some of the back-ends could actually implement this).

[ghost]

Thanks!

The context in which I was looking into this was from a jvm-based code, so direct library reference was a no-go. What i ended up doing was faking a direct library reference using a sidecar pattern in k8s. worked well enough, though a service would have been a nicer solution.