search

redis / node-redis

Redis Node.js client
https://redis.js.org/
MIT License
16.95k stars 1.89k forks source link

Pub/sub subscription does not resubscribe successfully after a socket closed unexpectedly error #2823

Open rodmccutcheon opened 3 months ago

rodmccutcheon commented 3 months ago

Description

We found when upgrading the node-redis client from version 3.1.2 to 4.7.0 it failed to recover after a socket closed unexpectedly error. We added all of the possible event listeners and we can see it attempting to reconnect, but never reach ready state:

Screenshot 2024-08-26 at 1 42 09 PM

This happens when the system is under significant load. This is affecting our ability to upgrade the node-redis client library in our production environment.

Production environment:

Node.js Version: "18.18.1" Redis Server Version: "6.2" Node Redis Version: "4.7.0" Platform: AWS Elasticache

Example code to reproduce: https://github.com/Milanote/node-redis-bug

I couldn't create a nice clean assertion, but the basic gist is this test case sends a large payload (5mb) frequently until the client-output-buffer-limit is reached. The v3 version seems to reconnect and resubscribe correctly and I can see more messages received. The v4 test case doesn't seem to be able to resubscribe correctly, and receives fewer messages overall. I've tried debugging the client code a bit, but found it hard to pinpoint the exact issue.

It seems to hang resolving the resubscribe promise: https://github.com/redis/node-redis/blob/942de1f0b4868f0f6464b2e0702b621a3373c4ee/packages/client/lib/client/commands-queue.ts#L324

Node.js Version

18.18.1

Redis Server Version

6.2

Node Redis Version

4.7.0

Platform

Linux

Logs

No response

leibale commented 2 months ago

I tried playing with your repo, and it seems like both versions are running about the same (the "happy case" works, and the 5MB case causes the connection to die -> reconnect -> resubscribe). Anyway, writing to the socket faster than the network/server can handle is something you should not do, and there is not much the client can do about it.. If you want we can debug this together, just ping me in the redis discord (my handle is @leibale)