Open Kaszanas opened 5 months ago
Hey @Kaszanas!
This library doesn't currently have support for setting up unlimited task queues. This was an intentional design decision for the sake of simplicity and efficiency. The task queue is currently implemented as a fixed-size buffered channel, which takes a constant space in terms of memory and minimizes the number of memory allocations when inserting or removing tasks. I guess there could be situations where it would make sense to have an unbounded task queue, but I see a few potential issues with that approach:
That said, I would encourage you to run some stress/load tests to determine a reasonable value for the maxCapacity
parameter (size of the buffered channel). I woudn't recommend using math.MaxInt
since that will cause the process to block the entire available RAM and crashing immediately :sweat_smile:.
You are right, these options could have been uint
instead of int
. I don't recall why I picked int
to be honest, maybe it had to do with minimizing the number of int conversions. It's probably too late to change this though, since that could break existing clients of this library :upside_down_face:
Thanks for the swift reply @alitto yeah I only wanted to have an unlimited task queue because it can get filled quicker than processing happens. In that case it could be hard to foresee what should be it's size.
Background
I have a system where I will be using pond for network related download tasks. To avoid hitting rate limiters when issuing multiple network requests it would be great to be able to set unlimited task capacity.
Imagine a scenario where there are some downloads that are already finished and the data is easily accessible so there is no need to issue another download.
In such case a broader worker pool can have as much goroutines as number of CPU threads. But then you wouldn't like to have that for a worker pool downloading because you'd have potentially 24 - 56 threads attempting that at once.
Question
Is there a way to have unlimited tasks queue and just 3-5 workers downloading at the same time? I was not able to find such example in the documentation and the code.
Would this mean that it should be set to the maximum integer capacity?
Int vs Uint?
Additionally given that these values cannot be lower than zero:
Wouldn't it be better to have these types be
uint
?