First thank you for your useful redis client that supports redis cluster, it helps us a lot.
I am experiencing unwanted behavior when connecting to Redis using this client in edge cases such as when the cluster is down. In such cases I would like to timeout the operation and throw an error, if for example my redis cluster has 6 nodes and the connection timeout is 1000ms and the connection attempts is 2, i would expect it to timeout after 2-3 seconds and throw the error. However, as stated the cluster is 6 nodes meaning that using this client, the above case will multiply by 6(nodes)*2(connection attempts)¸ at the best case and in some cases I saw that with 6 nodes and the above configuration there were 14 timeout errors of connections to the redis cluster, each taking ~1second, and a few host is unreachable errors, leading to the whole operation to take 20seconds+
See this example log:
msg:2021/06/21 16:06:24 [error] 61#0: *27289 [lua] rediscluster.lua:218: fetch_slots(): failed to fetch slots: timeout;timeout;timeout;timeout;timeout;timeout;timeout;timeout;timeout;timeout;host is unreachable;host is unreachable;timeout;timeout;timeout;timeout;host is unreachable;host is unreachable, client: 127.0.0.1, server: kong, request: "GET /mickeys-receiver/health HTTP/1.1", host: "mickey.co"
To solve this, I suggest adding a general (optional) configuration param called max_connection_timeout which will timeout the try_hosts_slots function after the configured amount (if exists), which will allow us to be more in control when we have latency or connection problems from our Redis cluster.
Would love to hear your thoughts about this. I have a PR ready for this, since we need this functionality in our deployment.
Hi @steve0511,
First thank you for your useful redis client that supports redis cluster, it helps us a lot.
I am experiencing unwanted behavior when connecting to Redis using this client in edge cases such as when the cluster is down. In such cases I would like to timeout the operation and throw an error, if for example my redis cluster has 6 nodes and the connection timeout is 1000ms and the connection attempts is 2, i would expect it to timeout after 2-3 seconds and throw the error. However, as stated the cluster is 6 nodes meaning that using this client, the above case will multiply by 6(nodes)*2(connection attempts)¸ at the best case and in some cases I saw that with 6 nodes and the above configuration there were 14 timeout errors of connections to the redis cluster, each taking ~1second, and a few host is unreachable errors, leading to the whole operation to take 20seconds+ See this example log:
To solve this, I suggest adding a general (optional) configuration param called
max_connection_timeoutwhich will timeout thetry_hosts_slotsfunction after the configured amount (if exists), which will allow us to be more in control when we have latency or connection problems from our Redis cluster.Would love to hear your thoughts about this. I have a PR ready for this, since we need this functionality in our deployment.
PR - https://github.com/steve0511/resty-redis-cluster/pull/85