search

redis / ioredis

🚀 A robust, performance-focused, and full-featured Redis client for Node.js.
MIT License
14.07k stars 1.19k forks source link

Failed requests on all shards while only a single shard failed #1803

Open khangly opened 10 months ago

khangly commented 10 months ago

Our service is on AWS ECS and our Redis is AWS ElastiCache (cluster mode enabled) with 3 shards. Each shard has a primary and a replica node, so 2 nodes for each shard. At 2023-08-08 07:00:00 UTC, a primary node failed so ElastiCache initiated a fail over. While that was happening, we had multiple failed write requests span across 3 shards uniformly, although it should only happen with the failed shard. During that time, new connections are also being made to all 3 shards as well, not just the failed shard.

This is our service configuration

{
  scaleReads: "slave",
  dnsLookup: (address, callback) => callback(null, address),
  slotsRefreshTimeout: 2000,
  clusterRetryStrategy(retries: number): number {
    return Math.min(retries * 100, 3000);
  },
  redisOptions: {
    username: "default",
    password,
    enableReadyCheck: true,
    tls: {},
    enableAutoPipelining: true,
    commandTimeout: 100,
    connectTimeout: 10000,
  },
  lazyConnect: true,
}

What could be the issue with our configs? Or is this expected behavior of ioredis?