redis / ioredis

🚀 A robust, performance-focused, and full-featured Redis client for Node.js.
MIT License
14.06k stars 1.19k forks source link

Async get is blocking #1849

Open ldechamps opened 5 months ago

ldechamps commented 5 months ago

We have a problem with the way ioRedis works. When retrieving a value via the get command from a stored value (Buffer type), there seems to be a blocking element at the asynchronous level (this effect is accentuated when values increase in size). For values of 2Mo , blocking times of 3ms can occur (less size, less blocking time but still blocking), which is worrying when Redis is used intensively. Is the problem known? Is there a solution? What might the problem be?

[14:27:54.658] INFO (20861): 595 - get test 
[14:27:54.661] INFO (20861): ping
[14:27:54.666] INFO (20861): 596 - get test
[14:27:54.668] INFO (20861): ping
[14:27:54.670] INFO (20861): 597 - get test 
[14:27:54.671] INFO (20861): ping
[14:27:54.674] INFO (20861): 598 - get test

Note : Ping is set to 1ms interval

Regards

ldechamps commented 5 months ago

I think I've found a clue: when retrieving the stream, the DataHandler uses redis-parser. At the end of the retrieval, it reconstitutes the entire data stream. This last operation may be blocking:

concatBulkBuffer

Is there a bypass? Can ioredis stream the get response directly?

ldechamps commented 3 months ago

I'm relaunching this thread. we've made little progress on this problem. Despite optimizing the data stream reconstruction, we still have blocking problems. And we realized that despite using the auto-pipeline mode, data recovery was sequential. In other words, if you retrieve cache1 followed by cache2 etc., then the last cache was retrieved much later, even though the command was sent at the same time as the others. Is there a way or a command to do them in parallel?

jcyh0120 commented 2 months ago

Facing this problem, too. I made a simple test by stream XREAD with block and then console.log the result. It has to wait until the XREAD return which block the whole process.