StackExchange / StackExchange.Redis

General purpose redis client
https://stackexchange.github.io/StackExchange.Redis/
Other
5.84k stars 1.5k forks source link

Invalid bulk string terminator #2674

Closed axelgenus closed 3 months ago

axelgenus commented 3 months ago

I am facing this exception when trying to read from a remote Redis server:

StackExchange.Redis.RedisConnectionException: Invalid bulk string terminator
   at StackExchange.Redis.ConnectionMultiplexer.ExecuteSyncImpl[T](Message message, ResultProcessor`1 processor, ServerEndPoint server, T defaultValue) in /_/src/StackExchange.Redis/ConnectionMultiplexer.cs:line 2109
   at StackExchange.Redis.RedisDatabase.Execute(String command, ICollection`1 args, CommandFlags flags) in /_/src/StackExchange.Redis/RedisDatabase.cs:line 1501
   at NRedisStack.JsonCommands.Get[T](RedisKey key, String path, JsonSerializerOptions serializerOptions)

I am using the official Debian Bullseye redis-stack package on the server, the latest version of StackExchange.Redis (2.7.33) and .NET 8. I have found no hints about what can cause the issue and with a local instance of Redis, the code seems to work just fine.

mgravell commented 3 months ago

What is the actual server here? Is it genuine redis?

If the problem is reproducible on demand, there is an API available that can capture your RESP traffic, allowing validation and investigation; please see https://stackexchange.github.io/StackExchange.Redis/RespLogging - are you able to see the issue with this enabled?

mgravell commented 3 months ago

TODO for self: switch to 100% isolated independent buffer inside the logging loop for the external read, to rule out any buffer reuse problem

axelgenus commented 3 months ago

What do you mean by "genuine Redis"? I am using the Debian package from the official Redis repository. The version is 7.2.0v9.

I am now checking with RESP logging enabled.

mgravell commented 3 months ago

That's a "yes" then - sometimes we've had problems with people using the library against other servers (AWS, Alibaba, etc) that implement the redis API, and we've seen those servers have glitches (especially with pipelined operations). Redis itself is usually pretty solid here!

axelgenus commented 3 months ago

I did not know there were custom re-implementations of the Redis API's; I just thought they were instances running in webfarms...

Anyways, I dug a bit into the issue:

I am going to keep investigating the logs.

axelgenus commented 3 months ago

I've just reproduced the issue. It seems the connection is severed by the server! I still do not know why but probably it's just a virtualization issue (virtio NIC), I guess it has nothing to do with the library. Sorry for wasting your time.