search

taskiq-python / taskiq

Distributed task queue with full async support
MIT License
866 stars 52 forks source link

Question about max prefetch #344

Closed realitix closed 2 months ago

realitix commented 4 months ago

I would like to know precisely how max_prefetch works. If I set workers = 6, for example, thus six worker processes, and I do not set max_prefetch, how will the task distribution among the 6 workers happen? Will a single worker retrieve all the tasks, or will the task distribution be balanced among the 6? For instance, if we have max_async_task set to 6, should max_prefetch also be set to 6 ?

Thanks

Sobes76rus commented 4 months ago

workers are spawned as child processes max_async_task and max_prefetch are set for each worker max_prefetch means how many messages (tasks) would be in queue awaiting for execution max_async_task + max_prefetch - maximum number of tasks that are already running (max_async_task) or waiting to be executed (max_prefetch) on a single worker

u dont need to set max_prefetch to 6 it's possible that one worker would retrieve all the tasks