search

traitecoevo / rrqueue

R + Redis queues
http://traitecoevo.github.io/rrqueue
Other
32 stars 9 forks source link

enqueue_bulk should talk to Redis in bulk (same with rrqlapply) #7

Open richfitz opened 9 years ago

richfitz commented 9 years ago

This would save on roundtrip communication and might not be that hard to implement. Main issues will be the back and forth with the id counter which is specially allocated.

richfitz commented 9 years ago

To build this, set up toxiproxy to simulate a slow Redis connection and test varying strategies. In particular, slow up/fast down would be a reasonable simulation of a home internet connection. Otherwise it's pretty much guesswork.

richfitz commented 9 years ago

See pipelining and scripting; there are a bunch of operations that could be done in a single round trip for each job that is submitted. That should see ~6x speed up in job submissions without having to do each job in bulk. I've done some lua scripting in RedisAPI or rrlite or nodbi to try mass insertion so that might be a good place to start