search

futureverse / future.apply

:rocket: R package: future.apply - Apply Function to Elements in Parallel using Futures
https://future.apply.futureverse.org
211 stars 16 forks source link

making stop() immediate #75

Open rickhelmus opened 3 years ago

rickhelmus commented 3 years ago

Hello,

Consider the following simple example:

library(future.apply)
plan(cluster, workers = 4)

future_lapply(1:4, function(x)
{
    if (x == 2)
        stop("x==2")
    Sys.sleep(10)
})

Ideally in my scenario this would stop immediately with an error message, but instead it takes ten seconds and then the error is thrown. Is there a way to change this? I saw a somewhat related discussion on throwing custom conditions, which I guess could also work, but to me the above situation seemed like a fairly common use case to have.

Thanks, Rick

HenrikBengtsson commented 3 years ago

Yes, this "early stopping" feature is frequently requested. In order to be able to do this, we need to be able to terminate the other three futures that were launched at the same time as the future that produced the error. Being able to terminate futures is something that is on the roadmap and it needs to be resolved before even being able to add that option to future_lapply() and friends.

FYI, the first place where this has to be implemented is in the parallelly package (https://github.com/HenrikBengtsson/parallelly/issues/33), then it'll have to be added to the future package (https://github.com/HenrikBengtsson/future/issues/93). Only when that is in place, it can be added to future.apply (or furrr, or doFuture). So, this is something that will take time. It's not something easy to fix but hopefully doable.

rickhelmus commented 3 years ago

Hi Henrik,

Thanks for the quick response! And thanks for the clear explanation on the current situation. Looking forward to when you get around to implement all these changes!