This is a PoC for reducing the Redis load caused by each worker frequently polling the entire set of subqueues.
SSCAN lets us prefilter the queues we pull out of Redis, and seems pretty fast too:
production In [49]: r.scard("t:scheduled")
production Out[49]: 3919
production In [50]: timeit.timeit(lambda: r.smembers("t:scheduled"), number=5)
production Out[50]: 0.06490249998751096
production In [51]: timeit.timeit(lambda: r.sscan("t:scheduled", match="zoom.*", count=10000), number=5)
production Out[51]: 0.01752800800022669
This is a PoC for reducing the Redis load caused by each worker frequently polling the entire set of subqueues.
SSCAN lets us prefilter the queues we pull out of Redis, and seems pretty fast too: