search

actionhero / node-resque

Node.js Background jobs backed by redis.
https://node-resque.actionherojs.com
Apache License 2.0
1.37k stars 151 forks source link

When using a Redis instance with a large keyspace, some operations can be extremely slow #325

Closed yjwong closed 4 years ago

yjwong commented 4 years ago

We have a Redis instance with approximately ~1.3 million keys. With this number of keys, anything that relies on connection.getKeys() (https://github.com/actionhero/node-resque/blob/master/src/core/connection.ts#L93), including queue.allDelayed(), queue.locks(), queue.stats() and even scheduler polling can get very slow, taking over 30 seconds. This is because redis.scan() gets invoked hundreds of thousands of times when connection.getKeys() gets called.

connection.getKeys() uses the SCAN command in Redis. When using SCAN, the COUNT parameter is set to 10 by default. Since the function is recursive, the stack depth can get really deep even with 10k keys. We have found success in increasing COUNT to a higher value, such as 1000, providing > 10-15x speedup in such a Redis instance. Would it make sense to increase this value in node-resque when calling scan()?

Default value (10): ~40s 500: ~5s 1000: ~3.75s 5000: ~3.2s 10000: ~2.9s

evantahler commented 4 years ago

That would be a great pull request to add, with a connection option to configure that COUNT!

evantahler commented 4 years ago

Let me know if you need help creating that pull request.

evantahler commented 4 years ago

Hello @yjwong! I'm checking in to see if you need help working on this pull request.

evantahler commented 4 years ago

Can you please evaluate https://github.com/actionhero/node-resque/pull/330 and let me know if that works for you?

evantahler commented 4 years ago

Solved by https://github.com/actionhero/node-resque/pull/330