search

redis / go-redis

Redis Go client
https://redis.uptrace.dev
BSD 2-Clause "Simplified" License
20.2k stars 2.38k forks source link

fix: Fixed issue with context cancelled error leading to connection spikes on Primary instances #3190

Open EXPEbdodla opened 1 week ago

EXPEbdodla commented 1 week ago

Issue: After upgrading from 9.5.1 to 9.7.0, we noticed the spikes in connections to Master/ Primary nodes and also reads are happening from Master nodes. Also noticed increase in pool_conn_total_current metrics.

Environment: AWS Cloud, Elasticache Redis with Cluster Mode

Cause: After debugging, we noticed that nodes are marked as failed when the context.Cancelled error is raised https://github.com/redis/go-redis/blob/930d904205691ff06104fcc3ac108177077def35/osscluster.go#L1324. This is introduced in v9.5.2.

We tested this by deploying the change from my Fork and noticed improvements. Elasticache Cluster Current Connections Screenshot: image

FYI: My first PR to go-redis. Happy to fix if any concerns.