privefl / bigstatsr

R package for statistical tools with big matrices stored on disk.
https://privefl.github.io/bigstatsr/
179 stars 30 forks source link

Allow for parallel registration outside of functions. #118

Closed privefl closed 2 years ago

privefl commented 4 years ago

E.g. if using ncores = NULL, use higher-level registration for parallel processing?

Could for example use

library(future)
plan(list(future.batchtools::batchtools_slurm, multisession))
doFuture::registerDoFuture()

to run one thing on many nodes on a Slurm cluster.

privefl commented 2 years ago

The confusing thing would be that some functions use OpenMP instead of {foreach} to parallelize, therefore this won't work for all functions.

I think I'll forget about this, unless there is a strong argument for it.