bpmapply was passing all the ... to all the workers, making its memory consumption quite bad
In detail, the wrap function was defined nested under the environment of the bpmapply method.
When the wrap function was serialized, it included the environment of its parents, including the environment of the bpmapply method itself, that contains the ... of all the iterations.
This effectively forced passing all the iterations to all the workers, increasing the memory consumption.
By defining the wrap function outside of the method we reduce the size of its environment and get much better performance.
bpmapply was passing all the ... to all the workers, making its memory consumption quite bad
In detail, the
wrapfunction was defined nested under the environment of the bpmapply method. When the wrap function was serialized, it included the environment of its parents, including the environment of the bpmapply method itself, that contains the ... of all the iterations.This effectively forced passing all the iterations to all the workers, increasing the memory consumption.
By defining the wrap function outside of the method we reduce the size of its environment and get much better performance.