StackExchange / StackExchange.Redis

General purpose redis client
https://stackexchange.github.io/StackExchange.Redis/
Other
5.89k stars 1.51k forks source link

LockTake failing #2518

Closed pianoman4873 closed 1 year ago

pianoman4873 commented 1 year ago

Hello, I am encountering a peculiar exception from a particular windows services accessing our Redis cluster and not from others - although they share the same Redis cluster configuration ( amongst additional IIS servers from other machines all sharing the same Redis cluster config ) -

StackExchange.Redis.RedisServerException: MISCONF Redis is configured to save RDB snapshots, but it's currently unable to persist to disk. Commands that may modify the data set are disabled, because this instance is configured to report errors during writes if RDB snapshotting fails (stop-writes-on-bgsave-error option). Please check the Redis logs for details about the RDB error.\r\n at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server, T defaultValue) in //src/StackExchange.Redis/ConnectionMultiplexer.cs:line 1986\r\n at StackExchange.Redis.RedisDatabase.LockTake(RedisKey key, RedisValue value, TimeSpan expiry, CommandFlags flags) in //src/StackExchange.Redis/RedisDatabase.cs:line 1437\r\n at

As mentioned, there's no such exception from any other service , although many of them execute write commands in general and the LockTake in particular abundantly.

Any idea what could be the reason for that ?

Thanks a lot !

PS This happened hundreds of times in the last week.

NickCraver commented 1 year ago

@pianoman4873 It sounds like maybe you have some nodes of a cluster out? This is a server-side error, it could be lack of disk access or disks running out of space or similar.

mgravell commented 1 year ago

I'm intrigued, but ultimately as Nick says: this error is coming from the server itself - there's literally nothing we can do to either cause or remedy this message. I would advise checking all the server nodes individually. Closing this out.

pianoman4873 commented 1 year ago

Thanks Nick and Marc.