socketio / socket.io-redis-streams-adapter

The Socket.IO adapter based on Redis Streams, allowing to broadcast events between several Socket.IO servers.
https://socket.io/docs/v4/redis-streams-adapter/
MIT License
30 stars 15 forks source link

"Error: timeout reached: only 0 responses received out of 1" while scaling up a high-traffic system with multiple nodes. #6

Open OnlySekai opened 1 year ago

OnlySekai commented 1 year ago

I encountered an error "Error: timeout reached: only 0 responses received out of 1" while scaling up a high-traffic system with multiple nodes.

My system run 2 same instances on digitalOceans, and sometime it had been crash with error:

/workspace/node_modules/@socket.io/redis-streams-adapter/dist/cluster-adapter.js:348 reject(new Error(timeout reached: only ${storedRequest.current} responses received out of ${storedRequest.expected})); Error: timeout reached: only 0 responses received out of 1 at listOnTimeout (node:internal/timers:559:17) at processTimers (node:internal/timers:502:7)

my setup redis `const redisClient = createClient({ url: config.REDIS_ADDRESS, });

const initRedis = () => { redisClient.on('error', err => console.log(err)); return redisClient.connect().then(() => console.log('connected redis success')); };`

my setup adapter is: Io.io = new Server(httpServer, { adapter: createAdapter(redisClient, { maxLen: 500_000, readCount: 5000, heartbeatInterval: 5_000, heartbeatTimeout: 120_000, }), cors: { origin: config.CORS_WEBSOCKET, }, connectionStateRecovery: { maxDisconnectionDuration: config.MAX_DISCONNECTION_DURATION, skipMiddlewares: true, }, }).use(checkTokenAuth);

my setup use witth express:

const server = http.createServer(app); const io = new Io(); Promise.all([db.init(), initRedis()]).then(async () => { server.listen(config.PORT, () => { logger.info(HttpNetwork is running at port: ${config.PORT}); }); io.startSocket(server, () => console.log(socket is running at port: ${config.PORT})); });

robertogallardo commented 1 year ago

I had a similar error when making a call to fetchSockets().

Error: timeout reached: only 0 responses received out of 1 at Timeout._onTimeout (node_modules/@socket.io/redis-streams-adapter/dist/cluster-adapter.js:348:28) at listOnTimeout (node:internal/timers:564:17) at process.processTimers (node:internal/timers:507:7) Error: timeout reached: only 0 responses received out of 1 at Timeout._onTimeout (node_modules/@socket.io/redis-streams-adapter/dist/cluster-adapter.js:348:28) at listOnTimeout (node:internal/timers:564:17) at process.processTimers (node:internal/timers:507:7)

The other instances show an error uncaughtException: Maximum call stack size exceeded in the hasBinary function of the util.js file.

My solution was to check for sockets following @darrachequesne's recommendation and not use fetchSockets. https://github.com/socketio/socket.io/issues/4183#issuecomment-982181865

mgarf commented 1 year ago

I've also been getting a similar error with const recipientsList = await io.in(exSocket.appGuid).fetchSockets();

timeout reached: only 3 responses received out of 7

hysapp commented 1 year ago

I have the same problem in version 0.1.0 when call let sockets = await io.fetchSockets() with 4 cpu using node js cluster

a lot of: timeout reached: only 0 responses received out of 4 . . . timeout reached: only 1 responses received out of 3 /root/daddy/node_modules/@socket.io/redis-streams-adapter/dist/cluster-adapter.js:348 reject(new Error(timeout reached: only ${storedRequest.current} responses received out of ${storedRequest.expected})); ^

Error: timeout reached: only 1 responses received out of 3 at Timeout._onTimeout (/root/daddy/node_modules/@socket.io/redis-streams-adapter/dist/cluster-adapter.js:348:28) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7)

Node.js v20.5.1 worker 1035162 died

introspection3 commented 1 year ago

@hysapp @OnlySekai hello,sirs,do you resolve this bug? I met this bug on cluster-adapter too . multiple nodes...

hysapp commented 1 year ago

@hysapp @OnlySekai hello,sirs,do you resolve this bug? I met this bug on cluster-adapter too . multiple nodes...

No, I implemented everything myself to be able to save and receive the list of online users, the stream adapter was not optimal at all and had bugs, and I used the sharded adapter.

introspection3 commented 1 year ago

@hysapp @OnlySekai hello,sirs,do you resolve this bug? I met this bug on cluster-adapter too . multiple nodes...

No, I implemented everything myself to be able to save and receive the list of online users, the stream adapter was not optimal at all and had bugs, and I used the sharded adapter.

can you tell me ,how to do it ,dear sir.

darrachequesne commented 8 months ago

https://github.com/socketio/socket.io-adapter/commit/0e23ff0cc671e3186510f7cfb8a4c1147457296f (included in version 0.2.0) should reduce the number of those errors.

@hysapp do you remember what kind of bugs you encountered?

introspection3 commented 8 months ago

I think this is not a bug ,but nodejs's event loop is blocked.

iakuf commented 2 weeks ago

/data/geelevel-signaling/node_modules/socket.io-adapter/dist/cluster-adapter.js:600 reject(new Error(timeout reached: missing ${storedRequest.missingUids.size} responses)); ^

Error: timeout reached: missing 1 responses at Timeout._onTimeout (/data/geelevel-signaling/node_modules/socket.io-adapter/dist/cluster-adapter.js:600:28) at listOnTimeout (node:internal/timers:581:17) at process.processTimers (node:internal/timers:519:7)

Node.js v22.5.1 I find other error , some one can help me?

iakuf commented 2 weeks ago

I try to used @socket.io/redis-adapter , not find this bug. only show in redis-streams-adapter.