In case of particularly large data sets the full number of processes distributed over several cluster nodes is only used for a short time. Then only a few processes on the master node are active and take up approx. the amount of memory defined by the function call plus another 10% for the master process. On smaller data sets the full number of available processes is used until no more jobs are pending.
Possibly the defined memory is seen as total for the whole process while it should be seen independently for each node. Furthermore each process should write its result to the disk individually, yet it seems like the results are first gathered by the master process/node and then written. Thus, because so much memory is already used on the master node, no more processes are started on the other available nodes.
What happens if the amount of memory is multiplied by the number of nodes? Would that solve the problem or is this an issue for the raster package? How to write results during execution time instead of watiting until the very end?
johntruckenbrodt
commented
6 years ago
In case of particularly large data sets the full number of processes distributed over several cluster nodes is only used for a short time. Then only a few processes on the master node are active and take up approx. the amount of memory defined by the function call plus another 10% for the master process. On smaller data sets the full number of available processes is used until no more jobs are pending. Possibly the defined memory is seen as total for the whole process while it should be seen independently for each node. Furthermore each process should write its result to the disk individually, yet it seems like the results are first gathered by the master process/node and then written. Thus, because so much memory is already used on the master node, no more processes are started on the other available nodes. What happens if the amount of memory is multiplied by the number of nodes? Would that solve the problem or is this an issue for the raster package? How to write results during execution time instead of watiting until the very end?