gresrun
commented
9 years ago We could look into a solution that caches the type of a queue to save all those calls.
As for your large queue vs small queues, I'd have to look at your workload. If the jobs are all short-lived and CPU-bound, It would make sense that only a few workers end up doing most of the work since those threads are actively running on the CPU and will get back to fetch the next job before other threads are woken up to get a job. Maybe you can tweak the number of worker threads or do some more investigation into it. I'd be curious to see what you find!
abhinavdwivedi
I am facing latency issues with Resque. I have dedicated a VM of Redis for Resque. Only resque workers and few locks that need to be taken during the job execution are done on this redis instance. Here is the command stats of that instance:
Commandstats
cmdstat_get:calls=2111,usec=10266,usec_per_call=4.86 cmdstat_set:calls=760626,usec=5961980,usec_per_call=7.84 cmdstat_setnx:calls=11379,usec=80801,usec_per_call=7.10 cmdstat_del:calls=769892,usec=4445869,usec_per_call=5.77 cmdstat_incr:calls=1521250,usec=11125827,usec_per_call=7.31 cmdstat_rpush:calls=630726,usec=19003304,usec_per_call=30.13 cmdstat_lpush:calls=630120,usec=3318287,usec_per_call=5.27 cmdstat_lpop:calls=1390349,usec=24332154,usec_per_call=17.50 cmdstat_llen:calls=12342,usec=53727,usec_per_call=4.35 cmdstat_lindex:calls=234718953,usec=740049726,usec_per_call=3.15 cmdstat_sadd:calls=762026,usec=4995570,usec_per_call=6.56 cmdstat_smembers:calls=1976,usec=177941,usec_per_call=90.05 cmdstat_zadd:calls=130905,usec=2291260,usec_per_call=17.50 cmdstat_zrem:calls=519398,usec=3726215,usec_per_call=7.17 cmdstat_zrangebyscore:calls=126509310,usec=1107421536,usec_per_call=8.75 cmdstat_zcard:calls=3952,usec=19675,usec_per_call=4.98 cmdstat_hset:calls=129110,usec=2005137,usec_per_call=15.53 cmdstat_hget:calls=129106,usec=1364512,usec_per_call=10.57 cmdstat_select:calls=7647,usec=29785,usec_per_call=3.89 cmdstat_expire:calls=9268,usec=58640,usec_per_call=6.33 cmdstat_ping:calls=816168,usec=2510628,usec_per_call=3.08 cmdstat_type:calls=1348568645,usec=5287065002,usec_per_call=3.92 cmdstat_multi:calls=233961325,usec=545398278,usec_per_call=2.33 cmdstat_exec:calls=233961325,usec=745074070,usec_per_call=3.18 cmdstat_info:calls=4,usec=3296,usec_per_call=824.00 cmdstat_config:calls=1,usec=483,usec_per_call=483.00 cmdstat_publish:calls=2,usec=55,usec_per_call=27.50 cmdstat_watch:calls=236251945,usec=691940962,usec_per_call=2.93 cmdstat_unwatch:calls=2290227,usec=10833822,usec_per_call=4.73
As you can see there are a 1.3 billion calls for getting the type of queue. I am using long lived queues whose type will not change. Can't we allow the client to pass type as well because it is an unnecessary call in my use case.
Quick question: Does jesque favor many small queues than few large ones, because when i benchmarked jesque i found that when queue is large only few workers pick and execute task(In this test my avg job execution time was 100ms and num of workers was 50... only 4-5 workers did 90% of 10K job).