redis / node-redis

Redis Node.js client
https://redis.js.org/
MIT License
16.93k stars 1.89k forks source link

SocketClosedUnexpectedlyError: Socket closed unexpectedly #2032

Closed wyvasi closed 2 years ago

wyvasi commented 2 years ago

Good day! The socket connection closes unexpectedly and doesn't reconnect, my older version of node-redis(2.8.0) was reconnecting fine and still does. After reconnect it doesn't print ready anymore. We are using AWS ElastiCache. I can only reproduce this on production env. We have a package that wraps node-redis and is used in all our apps, so I don't think is related to version difference. I tried to comment out next code and it doesn't close the connection at all (from @socket.io/redis-adapter).

this.subClient.pSubscribe(this.channel + "*", (msg, channel) => {
    this.onmessage(null, channel, msg);
}, true);
Redis connected
Redis ready!
Redis connected
Redis ready!
fetch error Error: timeout reached while waiting for fetchSockets response
    at Timeout._onTimeout (C:\work\test-socket\node_modules\@socket.io\redis-adapter\dist\index.js:615:28)
    at listOnTimeout (node:internal/timers:568:17)
    at processTimers (node:internal/timers:510:7)
List with sockets connected to hello room: 0
List with sockets connected to hello room: 5
SocketClosedUnexpectedlyError: Socket closed unexpectedly
    at Socket.<anonymous> (C:\work\test-socket\node_modules\@node-redis\client\dist\lib\client\socket.js:195:118)
    at Object.onceWrapper (node:events:510:26)
    at Socket.emit (node:events:390:28)
    at TCP.<anonymous> (node:net:687:12)
Redis reconnecting
Redis connected
ParserError: Protocol error, got "n" as reply type byte
    at handleError (C:\work\test-socket\node_modules\redis-parser\lib\parser.js:190:15)
    at parseType (C:\work\test-socket\node_modules\redis-parser\lib\parser.js:304:14) {
  offset: 304,
  buffer: '....'

Here is where I catch the parser error: client/socket.ts

if (this.#initiator) {
            try {
                await this.#initiator();
            } catch (err) {

List with buffers and offsets:

[{
        offset: 405,
        buffer: '{"type":"Buffer","data":[42,52,13,10,36,56,13,10,112,109,101,115,115,97,103,101,13,10,36,49,50,13,10,105,110,116,101,114,97,99,116,35,47,35,42,13,10,36,54,52,13,10,105,110,116,101,114,97,99,116,35,47,35,104,105,115,82,111,111,109,114,104,122,78,49,49,74,80,98,111,113,107,54,120,78,103,80,52,100,121,71,102,106,84,120,75,86,115,121,98,88,106,107,108,56,56,45,50,66,119,76,65,112,73,99,35,13,10,36,49,53,53,13,10,147,166,119,65,102,98,82,66,131,164,116,121,112,101,2,164,100,97,116,97,146,173,110,111,116,105,102,105,99,97,116,105,111,110,115,130,165,99,111,117,110,116,205,5,184,164,116,121,112,101,181,110,111,116,105,102,105,99,97,116,105,111,110,115,95,99,111,117,110,116,101,114,163,110,115,112,161,47,131,165,114,111,111,109,115,145,217,52,104,105,115,82,111,111,109,114,104,122,78,49,49,74,80,98,111,113,107,54,120,78,103,80,52,100,121,71,102,106,84,120,75,86,115,121,98,88,106,107,108,56,56,45,50,66,119,76,65,112,73,99,166,101,120,99,101,112,116,144,165,102,108,97,103,115,128,13,10,42,52,13,10,36,56,13,10,112,109,101,115,115,97,103,101,13,10,36,49,50,13,10,105,110,116,101,114,97,99,116,35,47,35,42,13,10,36,54,52,13,10,105,110,116,101,114,97,99,116,35,47,35,104,105,115,82,111,111,109,100,76,57,116,118,98,120,51,84,118,52,114,56,87,114,109,56,83,78,89,50,71,53,110,98,80,114,97,112,85,70,116,71,50,107,74,76,116,90,45,50,66,57,105,85,35,13,10,36,49,53,51,13,10,147,166,119,65,102,98,82,66,131,164,116,121,112,101,2,164,100,97,116,97,146,173,110,111,116,105,102,105,99,97,116,105,111,110,115,130,165,99,111,117,110,116,110,164,116,121,112,101,181,110,111,116,105,102,105,99,97,116,105,111,110,115,95,99,111,117,110,116,101,114,163,110,115,112,161,47,131,165,114,111,111,109,115,145,217,52,104,105,115,82,111,111,109,100,76,57,116,118,98,120,51,84,118,52,114,56,87,114,109,56,83,78,89,50,71,53,110,98,80,114,97,112,85,70,116,71,50,107,74,76,116,90,45,50,66,57,105,85,166,101,120,99,101,112,116,144,165,102,108,97,103,115,128,13,10,42,52,13,10,36,56,13,10,112,109,101,115,115,97,103,101,13,10,36,49,50,13,10,105,110,116,101,114,97,99,116,35,47,35,42,13,10,36,54,50,13,10,105,110,116,101,114,97,99,116,35,47,35,104,105,115,82,111,111,109,120,89,112,113,90,54,101,111,89,48,107,120,90,105,81,86,113,97,111,89,90,99,69,119,121,72,50,100,70,122,78,115,108,52,82,50,114,122,111,109,85,113,107,35,13,10,36,49,53,49,13,10,147,166,119,65,102,98,82,66,131,164,116,121,112,101,2,164,100,97,116,97,146,173,110,111,116,105,102,105,99,97,116,105,111,110,115,130,165,99,111,117,110,116,113,164,116,121,112,101,181,110,111,116,105,102,105,99,97,116,105,111,110,115,95,99,111,117,110,116,101,114,163,110,115,112,161,47,131,165,114,111,111,109,115,145,217,50,104,105,115,82,111,111,109,120,89,112,113,90,54,101,111,89,48,107,120,90,105,81,86,113,97,111,89,90,99,69,119,121,72,50,100,70,122,78,115,108,52,82,50,114,122,111,109,85,113,107,166,101,120,99,101,112,116,144,165,102,108,97,103,115,128,13,10,42,52,13,10,36,56,13,10,112,109,101,115,115,97,103,101,13,10,36,49,50,13,10,105,110,116,101,114,97,99,116,35,47,35,42,13,10,36,54,50,13,10,105,110,116,101,114,97,99,116,35,47,35,104,105,115,82,111,111,109,53,111,74,87,66,49,70,85,50,108,109,86,73,54,55,70,106,68,99,83,103,71,118,121,68,48,76,66,83,90,67,83,97,67,113,103,97,69,115,86,119,120,85,35,13,10,36,49,53,50,13,10,147,166,119,65,102,98,82,66,131,164,116,121,112,101,2,164,100,97,116,97,146,173,110,111,116,105,102,105,99,97,116,105,111,110,115,130,165,99,111,117,110,116,204,203,164,116,121,112,101,181,110,111,116,105,102,105,99,97,116,105,111,110,115,95,99,111,117,110,116,101,114,163,110,115,112,161,47,131,165,114,111,111,109,115,145,217,50,104,105,115,82,111,111,109,53,111,74,87,66,49,70,85,50,108,109,86,73,54,55,70,106,68,99,83,103,71,118,121,68,48,76,66,83,90,67,83,97,67,113,103,97,69,115,86,119,120,85,166,101,120,99,101,112,116,144,165,102,108,97,103,115,128,13,10,42,52,13,10,36,56,13,10,112,109,101,115,115,97,103,101,13,10,36,49,50,13,10,105,110,116,101,114,97,99,116,35,47,35,42,13,10,36,54,52,13,10,105,110,116,101,114,97,99,116,35,47,35,104,105,115,82,111,111,109,53,85,74,71,49,53,87,110,116,103,79,88,115,116,106,115,49,45,50,66,82,66,76,106,72,102,82,68,65,109,115,122,112,98,114,82,86,82,69,102,73,50,105,53,85,35,13,10,36,49,53,52,13,10,147,166,119,65,102,98,82,66,131,164,116,121,112,101,2,164,100,97,116,97,146,173,110,111,116,105,102,105,99,97,116,105,111,110,115,130,165,99,111,117,110,116,204,155,164,116,121,112,101,181,110,111,116,105,102,105,99,97,116,105,111,110,115,95,99,111,117,110,116,101,114,163,110,115,112,161,47,131,165,114,111,111,109,115,145,217,52,104,105,115,82,111,111,109,53,85,74,71,49,53,87,110,116,103,79,88,115,116,106,115,49,45,50,66,82,66,76,106,72,102,82,68,65,109,115,122,112,98,114,82,86,82,69,102,73,50,105,53,85,166,101,120,99,101,112,116,144,165,102,108,97,103,115,128,13,10]}'
    },
    {
        offset: 136,
        buffer: '{"type":"Buffer","data":[42,51,13,10,36,57,13,10,115,117,98,115,99,114,105,98,101,13,10,36,49,57,13,10,105,110,116,101,114,97,99,116,45,114,101,113,117,101,115,116,35,47,35,13,10,58,49,13,10,42,51,13,10,36,57,13,10,115,117,98,115,99,114,105,98,101,13,10,36,50,48,13,10,105,110,116,101,114,97,99,116,45,114,101,115,112,111,110,115,101,35,47,35,13,10,58,50,13,10,42,51,13,10,36,57,13,10,115,117,98,115,99,114,105,98,101,13,10,36,50,55,13,10,105,110,116,101,114,97,99,116,45,114,101,115,112,111,110,115,101,35,47,35,88,115,52,75,106,79,35,13,10,58,51,13,10,42,51,13,10,36,57,13,10,115,117,98,115,99,114,105,98,101,13,10,36,50,52,13,10,105,110,116,101,114,97,99,116,45,114,101,113,117,101,115,116,35,47,97,100,109,105,110,35,13,10,58,52,13,10,42,51,13,10,36,57,13,10,115,117,98,115,99,114,105,98,101,13,10,36,50,53,13,10,105,110,116,101,114,97,99,116,45,114,101,115,112,111,110,115,101,35,47,97,100,109,105,110,35,13,10,58,53,13,10,42,51,13,10,36,57,13,10,115,117,98,115,99,114,105,98,101,13,10,36,51,50,13,10,105,110,116,101,114,97,99,116,45,114,101,115,112,111,110,115,101,35,47,97,100,109,105,110,35,104,65,112,66,90,76,35,13,10,58,54,13,10,42,51,13,10,36,49,48,13,10,112,115,117,98,115,99,114,105,98,101,13,10,36,49,50,13,10,105,110,116,101,114,97,99,116,35,47,35,42,13,10,58,55,13,10,42,51,13,10,36,49,48,13,10,112,115,117,98,115,99,114,105,98,101,13,10,36,49,55,13,10,105,110,116,101,114,97,99,116,35,47,97,100,109,105,110,35,42,13,10,58,56,13,10]}'
    },
    {
        offset: 12,
        buffer: '{"type":"Buffer","data":[42,52,13,10,36,56,13,10,112,109,101,115,115,97,103,101,13,10,36,49,50,13,10,105,110,116,101,114,97,99,116,35,47,35,42,13,10,36,54,56,13,10,105,110,116,101,114,97,99,116,35,47,35,104,105,115,82,111,111,109,90,71,86,107,121,121,84,115,98,77,84,45,50,66,111,76,53,69,104,82,106,69,121,81,78,116,48,100,69,57,99,66,52,45,50,70,54,67,90,76,113,81,113,45,50,66,117,81,119,35,13,10,36,49,53,56,13,10,147,166,119,65,102,98,82,66,131,164,116,121,112,101,2,164,100,97,116,97,146,173,110,111,116,105,102,105,99,97,116,105,111,110,115,130,165,99,111,117,110,116,204,131,164,116,121,112,101,181,110,111,116,105,102,105,99,97,116,105,111,110,115,95,99,111,117,110,116,101,114,163,110,115,112,161,47,131,165,114,111,111,109,115,145,217,56,104,105,115,82,111,111,109,90,71,86,107,121,121,84,115,98,77,84,45,50,66,111,76,53,69,104,82,106,69,121,81,78,116,48,100,69,57,99,66,52,45,50,70,54,67,90,76,113,81,113,45,50,66,117,81,119,166,101,120,99,101,112,116,144,165,102,108,97,103,115,128,13,10]}'
    },
    {
        offset: 88,
        buffer: '{"type":"Buffer","data":[42,52,13,10,36,56,13,10,112,109,101,115,115,97,103,101,13,10,36,49,50,13,10,105,110,116,101,114,97,99,116,35,47,35,42,13,10,36,54,56,13,10,105,110,116,101,114,97,99,116,35,47,35,104,105,115,82,111,111,109,89,111,97,115,90,54,72,53,80,45,50,66,71,45,50,66,80,45,50,66,80,49,90,106,82,87,74,87,87,83,54,113,72,74,50,86,120,80,77,68,78,51,117,81,101,120,75,87,81,35,13,10,36,49,53,56,13,10,147,166,104,55,52,78,66,121,131,164,116,121,112,101,2,164,100,97,116,97,146,173,110,111,116,105,102,105,99,97,116,105,111,110,115,130,165,99,111,117,110,116,204,194,164,116,121,112,101,181,110,111,116,105,102,105,99,97,116,105,111,110,115,95,99,111,117,110,116,101,114,163,110,115,112,161,47,131,165,114,111,111,109,115,145,217,56,104,105,115,82,111,111,109,89,111,97,115,90,54,72,53,80,45,50,66,71,45,50,66,80,45,50,66,80,49,90,106,82,87,74,87,87,83,54,113,72,74,50,86,120,80,77,68,78,51,117,81,101,120,75,87,81,166,101,120,99,101,112,116,144,165,102,108,97,103,115,128,13,10]}'
    }]

Code for creating the client

const Redis = require('redis');
Redis.debug_mode = true;

const getClient = async (url) => {

    const client = Redis.createClient({
        socket: {
            host: url
            port: 6379,
        }
    });
    client.on('error', (err) => {
        console.error(err);
    });
    client.on('connect', () => {
        console.log('Redis connected');
    });
    client.on('reconnecting', () => {
        console.log('Redis reconnecting');
    });
    client.on('ready', () => {
        console.log('Redis ready!');
    });

    await client.connect();
    return client;
};

module.exports = {
    getClient,
};

Environment:

leibale commented 1 year ago

@niksy https://github.com/redis/node-redis/issues/1993#issuecomment-1558774230

ishaarawy commented 1 year ago

changing from socket connection to url solved the issue on local machine connecting to docker container

redisClient = redis.createClient({
          url: 'redis://127.0.0.1:6379'
        });
djdabs commented 1 year ago

not sure if related but I'm getting similar errors. All event listeners are configured. Application is a nestjs server dockerized on node:18-alpine, talking to memorystore redis in GCP (idle connections are not automatically closed).

client initially connects -> socket gets closed -> client tries reconnecting -> uncaught exception occurs for socket closing

    "redis": "^4.6.7",
 async connectToRedis(): Promise<void>{
    try{
      const pubClient = createClient({
          socket: {
                host: process.env.REDIS_HOST,
                port: Number(process.env.REDIS_PORT)
         },
      });

      const subClient = pubClient.duplicate();

      pubClient.on('error', (error) => {
        console.error('pubClient redis error:', error);
      });
      pubClient.on('connect', () => console.log('pubClient redis is connected'));
      pubClient.on('reconnecting', () => console.log('pubClient redis is reconnecting'));
      pubClient.on('ready', () => console.log('pubClient redis is ready'));

      subClient.on('error', (error) => {
        console.error('subClient redis error:', error);
      });
      subClient.on('connect', () => console.log('subClient redis is connected'));
      subClient.on('reconnecting', () => console.log('subClient redis is reconnecting'));
      subClient.on('ready', () => console.log('subClient redis is ready'));

      await pubClient.connect()
      await subClient.connect()

      this.adapterConstructor = createAdapter(pubClient, subClient);
    } catch(err){
      console.log(err)
    }

  }

CleanShot 2023-06-02 at 14 59 36

niksy commented 1 year ago

@djdabs try this: https://github.com/redis/node-redis/issues/2443#issuecomment-1573393354

djdabs commented 1 year ago

@djdabs try this: #2443 (comment)

Still getting a triggerUncaugthException after client connects -> attempts to reconnect, but I'm now getting a different error.

fwiw, i don't think i need a pingInterval with GCP. Their docs say by default no idle timeout is configured. https://cloud.google.com/memorystore/docs/redis/supported-redis-configurations#modifiable_configuration_parameters

Error: write EPIPE
    at afterWriteDispatched (node:internal/stream_base_commons:160:15)
    at writeGeneric (node:internal/stream_base_commons:151:3)
    at Socket._writeGeneric (node:net:930:11)
    at Socket._write (node:net:942:8)
    at doWrite (node:internal/streams/writable:411:12)
    at clearBuffer (node:internal/streams/writable:572:7)
    at Writable.uncork (node:internal/streams/writable:351:7)
    at Immediate._onImmediate (/usr/src/app/node_modules/@redis/client/dist/lib/client/socket.js:84:69)
    at process.processImmediate (node:internal/timers:476:21) {

CleanShot 2023-06-05 at 11 37 28

djdabs commented 1 year ago

resolved my issue.. after switching to ioredis the error message got more specific and I was able to track down the underlying problem (redis required TLS). Disabled TLS on memorystore and it was able to work now.

redis error: ParserError: Protocol error, got "\u0015" as reply type byte. Please report this

stefanodecillis commented 1 year ago

Are there any updates on this issue? I'm currently facing the same problem while using a Redis instance served by Digital Ocean. TLS is in place and the job is listening on "error" - each time the issue occurs, I reset the connection. I don't know if it is the best solution but it kind of works even if it throws the error every five minutes.

In the previous comments, there is a suggestion that pings the server. Do you think that is the best way to handle it at the moment? Or should we wait the patch?

jorenvandeweyer commented 1 year ago

@stefanodecillis You can just use pingInterval since #2524 is fixed in version redis@4.6.8

shobhitbehl98 commented 10 months ago

Getting the same error at v4.6.12 `const redis = require('redis'); require('dotenv').config(); const client = redis.createClient({ url: process.env.KV_URL }); (async () => { await client.connect(); })(); client.on('connect', () => { console.log('Connected to Redis'); });

client.on('error', (err) => { console.error(Redis Error: ${err}); });

client.on('reconnecting', () => console.log('client is reconnecting')); client.on('ready', () => console.log('client is ready'));

module.exports = client;`

This is the error im gettting

ERROR Redis Error: Error: Socket closed unexpectedly INFO client is reconnecting INFO Connected to Redis ERROR Redis Error: Error: Socket closed unexpectedly ERROR Redis Error: Error: Socket closed unexpectedly INFO client is reconnecting INFO Connected to Redis ERROR Redis Error: Error: Socket closed unexpectedly ERROR Redis Error: Error: Socket closed unexpectedly Task timed out after 10.02 seconds

It runs well on local but not when deployed on vercel

jake-chambers commented 10 months ago

We are experiencing the same issue. We're listening error events explicitly...our connection logic is below @leibale :

export const createRedisClient = (options: RedisClientOptions) => {
    const client = createClient(options);
    client.on('error', async error => {
        console.error(`Redis client error : ${error}`);
        Sentry.captureException(error); // Capture the error with Sentry

        if (
            error.code === 'ECONNRESET' ||
            error.code === 'EPIPE' ||
            error.code === 'ECONNREFUSED'
        ) {
            // Disconnect before attempting to reconnect
            await client.disconnect();

            client.connect().catch(err => {
                console.error(`Failed to reconnect redis client: ${err}`);
                Sentry.captureException(err); // Capture the reconnection error with Sentry
            });
        }
    });
    client.connect().catch(err => {
        console.error(`Failed to connect redis client: ${err}`);
        Sentry.captureException(err); // Capture the reconnection error with Sentry
    });
    return client;
};
arpit-absyadav commented 9 months ago

The error message indicates that Redis is running in protected mode due to enabled protected mode and the absence of a password for the default user. In this mode, connections are only accepted from the loopback interface. To enable connections from external computers, you can choose one of the following solutions:

Disable protected mode by executing the command: 'CONFIG SET protected-mode no'.