search

istio-ecosystem / authservice

Move OIDC token acquisition out of your app code and into the Istio mesh
Apache License 2.0
217 stars 63 forks source link

Redis error during high traffic times #233

Open brostwalt87 opened 1 year ago

brostwalt87 commented 1 year ago

We are currently running authservice using AWS Elasticache as a backing store. During high traffic hours, we often see this issue in the logs

[2023-03-02 15:14:44.482] [console] [trace] hmget: redis connection closed error, retrying
[2023-03-02 15:14:44.482] [console] [trace] hmget: redis connection closed error, retrying
[2023-03-02 15:14:44.483] [console] [trace] hmget: redis connection closed error, retrying
[2023-03-02 15:14:44.483] [console] [error] hmget: redis connection closed error, throwing error

I have looked into different ways of tweaking the Elasticache server, including upgrading the instance to cache.m5.large. So pure resource consumption does not seem to be an issue. The elasticache server is running 2 replicas - a primary and secondary. Also, authservice is running a deployment with 2 pods, but has the ability to autoscale to 3. Resources do not seem to be an issue there

Also we have the following set on redis/elasticache: appendonly no maxmemory-policy allkeys-lru

Would like to see if there is a way to either optimize redis or increase the number of retries via a variable that can be added in at runtime