gresrun / jesque

An implementation of Resque in Java.
http://gresrun.github.io/jesque
Apache License 2.0
628 stars 131 forks source link

Add functionality to enqueue jobs in batches #153

Closed maciejp closed 5 years ago

maciejp commented 5 years ago

We are using Jesque to enqueue a lot (up to 100 000) of jobs at once. Doing it in a normal, sequential way is not performant enough - we observed an average enqueuing time of 1ms/job. This can be significantly improved by either using pipelininig or transactions feature of Redis. Since it is easier to implement the batching functionality using transactions, as there is no need to handle situations in which only a subset of jobs has been successfully scheduled, this is what this implementation is using. Note: Transactions are handled by Redis v1.2 and higher. Testing it locally, it took 16s to enqueue 10k jobs one by one and 400ms in batching mode.

This implementation does not take care of splitting the batch into chunks of 10k, or less, queries (as advised in the document above). This might potentially lead to increased memory use on Redis.

closes #123

coveralls commented 5 years ago

Coverage Status

Coverage increased (+0.07%) to 70.348% when pulling 2b61e9db0b839ee8eaa10b743759756ae7dcc25c on maciejp:implement-batch-enqueueing into 88bab76606af3270db630eb38c34d95bf018976f on gresrun:master.

maciejp commented 5 years ago

Closing for now as my understanding of Redis TX was incorrect.