Is there a way to retain the content of .future/${job} once a job is finished

[statquant]

Hello, please pardon the trivial question but somehow I did not find the answer from the docs in batchtools or doFuture vignettes. I am using a slurm cluster and when retrieving results of the workers I am getting some errors on some workers. I've noticed that a .future directory is created when the tasks are worked but is then "cleaned" when the job is finished. Is there a way to retain the .future directory so I can look at the log of the problematic tasks?

For reference I use the following code, and typically in the res_list result a few elements would be errors objects

res_list<- foreach(run_date = run_dates,  .errorhandling = "pass") %dopar% 
{                                                                           
    flog.info("Running  date %s", Ymd(run_date))         
    res <- run_fun(run_date)
}

Thank you very much for your help and your package

[HenrikBengtsson]

There's an open feature request for this in future.batchtools - https://github.com/HenrikBengtsson/future.batchtools/issues/37. When this is implemented and the new version is released, you'll be able to set options(future.delete = FALSE) to prevent the automatic cleanup.

[statquant]

I see, do you want a push request for this or is it done already ? Regards

[HenrikBengtsson]

Thanks for the offer - I needed to check in what was needed to be done first. I suspected there would be a need to move things around between different local functions, but turned out to be much easier than I though. I've updated future.batchtools so that options(future.delete = FALSE) is now fully respected. Install as:

remotes::install_github("HenrikBengtsson/future.batchtools@develop")

until this in on CRAN.