mtmorgan
commented
4 months ago We try to explain this on the help page for, e.g., ?MulticoreParam under the tasks: argument, e.g.,
A 'tasks' value of > 0 specifies the exact number of tasks.
Values can range from 1 (all of 'X' to a single worker) to
the length of 'X' (each element of 'X' to a different
worker).With 3 workers and 7 tasks for a vector of length 7, tasks 1, 2, 3 are sent to workers 1, 2, 3. When any of those, e.g., task 3, finishes, task 4 is sent, then 5, .... This is a 'round robin' and provides an effective way of parallelizing tasks whose completion time is unknown -- workers are always busy until all tasks have been assigned.
f = function(i) { Sys.sleep(3 - (i - 1) %% 3); message(i); i }
x = bplapply(1:7, f, BPPARAM = MulticoreParam(tasks = 7))
## 3
## 6
## 2
## 5
## 1
## 4
## 7
z5ouyang
This is more of a question which I cannot find answers anywhere. Please do share a link if it is answered some where. It seems if tasks is set to the length of X, the parallel is done in batches? Is the strategy like this: If X is 1:9, and set workers=3 and tasks=9, parallel will start with 1, 2, 3, until all 1, 2, 3 completed, then 4,5,6 will start, and until all 4,5,6 completed, 7,8,9 will start.
Am I understanding this correctly? I originally thought (isn't this more efficient?), it would start with 1,2,3, if 2 completed, 4 would start, and the if 3 completed, 5 would start, and if 4 completed, 6 would start, etc .. (This seems not the case)
Thank you!