keller-mark / pizzarr

Slice into Zarr arrays in R 🍕
https://keller-mark.github.io/pizzarr/
Other
29 stars 3 forks source link

pbapply `cl` argument can't be an integer when using windows #105

Open dblodgett-usgs opened 4 months ago

dblodgett-usgs commented 4 months ago

Describe the bug From pbapply docs:

A cluster object created by makeCluster, or an integer to indicate number of child-processes (integer values are ignored on Windows) for parallel evaluations (see Details on performance).

I'll put in a little work around, but we should probably figure out how to allow a person to pass in their own cluster object rather than using an integer number of workers.

There is a bit of overhead to spin up a cluster and shut it down. With future, the cluster gets created a-priori, which is akin to dask or the cluster from parallel. I think it's probably going to be best to accept that kind of cluster input in some way in addition to this simple setting of the level of parallelism on a per call basis.