StackExchange / StackExchange.Redis

General purpose redis client
https://stackexchange.github.io/StackExchange.Redis/
Other
5.86k stars 1.5k forks source link

Connect standalone cluster (1 node, no sentinel, standalone) CLUSTER command passed #2569

Closed mscbpi closed 9 months ago

mscbpi commented 10 months ago

Hi,

We are using Redis Bitnami helmchart to deploy a non-cluster, single-node, no sentinel, in k8s, standalone architecture. Clients are running in the same K8s cluster and accessing with an internal ClusterIP service.

We are using a standard, minimal connection string. <clusterIP URL>:6379,password=<password>,ssl=False,abortConnect=False,syncTimeout=7100

INFO Errorstats is reporting lots of errors (errorstat_ERR, several 100Ks), and we struggle to identify what they are.

We have identified CLUSTER NODES command that triggers an ERR that is expected that is regularly passed to the server ERR This instance has cluster support disabled

which would be expected.

Is there something we miss to connect this ? Is excluding CLUSTER command the right way to avoid this counter to grow ? Is there a way to identify the numerous ERR we have in the counters (there are certainly more) ?

Thanks

mgravell commented 10 months ago

We use cluster in part to find out whether it is in cluster mode. You should be able to add ,$cluster= to your connection string to disable this

mgravell commented 10 months ago

(basically, we pipeline everything we might possibly need when connecting, to avoid having to pay N x {latency} overhead)

mscbpi commented 10 months ago

@mgravell thanks a lot,

Great, that's easy to be done, we wanted to be sure it was the right way to avoid the CLUSTER command.

But is it normal that it would do that dozens of time per minutes ? (we have half a dozen container that are redis client running StackExchange.Redis) or does this imply a wrongdoing on the client side / connection management ?

Any hints how can I track the numerous other ERR that is counted on my redis server when making a call from StackExchange.Redis client ?

apart from this one there are others I could not detect, and I cannot see anything in my client.

kubectl exec -it -n <namespace> pods/redis-master-0 -- redis-cli -a <password> INFO | grep error

unexpected_error_replies:0
total_error_replies:167774
errorstat_ERR:count=167774

And I reset counters just a while ago. ,$cluster= still has to be added but counting CLUSTER NODES command does not match the counter above (while being part of it, obviously).

mscbpi commented 10 months ago

Subsidiary question:

Considering an app that implement Redis Client, single instance of the app, doing one Connect() call as per documented, how many clients am I supposed to see on Redis server during the whole lifecycle of the app (and many set/get command to store/retrieve keys in db ?)

I read 2 (when using pub/sub), but I have hundreds in CLIENT LIST suspecting there is something wrong with connection handling.

mscbpi commented 9 months ago

Investigating the code, we had indeed a wrongdoing in managing the connection.

2 per Redis Client is what is expected, one of we exclude SUBSCRIBE if not used.

The "store and reuse" ConnectionMultiplexer had not been respected.

I close the issue as ERR (and certainly other side effects) are related to CLUSTER commands that were triggered at each caching operation (GET or SET) because of Connect() that should have been avoided.

mgravell commented 9 months ago

@NickCraver do we currently track how many multiplexers have been created? Maybe we should, and include in the error message if it is more than, say, 100: "hey, it looks like you're creating lots of multiplexers... that sounds wrong"

mscbpi commented 9 months ago

Great idea,

And is there a way to log/view the replies / errors from redis server to redis client ?

Thanks