DavisVaughan / furrr

Apply Mapping Functions in Parallel using Futures
https://furrr.futureverse.org/
Other
699 stars 40 forks source link

How to make future_walk() non-blocking? #178

Closed privefl closed 4 years ago

privefl commented 4 years ago

I would like to use packages {future.batchtools} and {furrr} to run loops in parallel on some HPC cluster. In my loops, I save each result in some RDS file, and basically return nothing. Is there any way to run future_walk() for this that would be non-blocking, i.e. once jobs for each element of the loop has been scheduled, then the R session becomes free again.

DavisVaughan commented 4 years ago

There is no way to do this with furrr, but you can use future::future() directly alongside a simple for loop to achieve this

I don't believe it makes sense for furrr to have a non blocking mode.