hibiken / asynq

Simple, reliable, and efficient distributed task queue in Go
MIT License
9.49k stars 688 forks source link

perf: Reuse rand for shuffling queues #916

Open l0nax opened 2 weeks ago

l0nax commented 2 weeks ago

Instead of creating a new rand.Rand every time we need the next queues, we create it once and reuse it.

The reason for this change comes from monitoring our application: After a few minutes, the github.com/hibiken/asynq.(*processor).queues method will consume a tremendous amount of CPU time and create a lot of garbage.

Since the queue method is called in its own goroutine, there is no need to add any synchronization primitives.

Here is the profiling flame graph of our application in a time window of 5 minutes: image