redis / ioredis

🚀 A robust, performance-focused, and full-featured Redis client for Node.js.
MIT License
14.31k stars 1.19k forks source link

Weird issue with connection dropping with cluster pub/sub #1772

Open argjent-dua opened 1 year ago

argjent-dua commented 1 year ago

Hello,

I'm using redis pub/sub to receive keyspace notifications for the expired keys. The solution works fine with the redis instance that doesn't have cluster mode enabled, but when switching to a cluster deployment it sometimes drops the connection and the subscriber won't reconnect.

I'm using Elasticache 7.0.7 Cluster Deployment with 2 shards & 2 nodes per shard. The output of debug logging for the subscriber is the following

2023-06-15T19:45:22.150Z ioredis:cluster:subscriber selected a subscriber xxx.xx.xx.17:6379
2023-06-15T19:45:22.151Z ioredis:cluster:subscriber started
2023-06-15T19:45:22.255Z ioredis:cluster:subscriber started
2023-06-15T19:45:22.255Z ioredis:cluster:subscriber selected a subscriber xxx.xx.xx.152:6379
2023-06-15T19:58:25.747Z ioredis:cluster:subscriber subscriber has disconnected, but ClusterSubscriber is not started, so not reconnecting.
2023-06-15T19:58:25.744Z ioredis:cluster:subscriber subscriber has disconnected, but ClusterSubscriber is not started, so not reconnecting.
2023-06-15T19:58:25.743Z ioredis:cluster:subscriber stopped
2023-06-15T19:58:25.730Z ioredis:cluster:subscriber stopped

The pub/sub subscriber is in a fargate task inside the same VPC as the redis instance.

The subscriber connects the following way:

new IORedis.Cluster(
        [
          {
            host,
            port,
          },
        ],
        {
          enableOfflineQueue: false,
          lazyConnect: true,
          redisOptions: {
                connectTimeout: 1000,
                enableOfflineQueue: false,
                autoResubscribe: true,
                maxRetriesPerRequest: 1,
                lazyConnect: true,
                retryStrategy(times: number) {
                  logger.warn(`Retrying redis connection: Attempt ${times}`);
                  return Math.min(times * 500, 20 * 1000);
                },
              },
        },
      );
 pubSubClient.psubscribe('__keyspace@*__:*', () => {});
 pubSubClient.on('pmessage', () => {});

I've also checked the maintenance of elasticache if it caused this, but I didn't find anything that happened. The instance was working normally as always. If this issue can't be solved how can I retry the whole subscription manually if this ever occurs?