Using future and memoise together

[Rekyt]

We officially recommended not to use memoise and future at the same time in the fundiversity manuscript. However, there maybe ways to get both.

I will collect here possibilities to work with both:

• A question on SO about using plumber, future, and memoise together: https://stackoverflow.com/q/70805314
• A discussion in the future repo: https://github.com/HenrikBengtsson/future/discussions/506
• An answer on SO (by Henrik Bengsston the author of future) suggesting to use R.cache file caching system instead: https://stackoverflow.com/a/48102804
• This open issue on memoise repo: https://github.com/r-lib/memoise/issues/29

It seems for now that using memoise it's not straightforward to parallelize. Maybe we should be extra-careful and add a warning when loading the package with memoisation that it shouldn't be used with parallelization. We should also add this in:

1. The README file
2. The parallelization vignette
3. The memoisation documentation segment
4. The parallelization documentation segment.

[Bisaloo]

Thanks for starting this discussion, this is helpful!

My intuition for R.cache is that it's not worth it in our case. Our tasks are still relatively fast and I'm not sure we can compensate the overhead of opening & writing to file. We can properly benchmark it at some point. I guess it will depend on how many times the memoised results are reused though and I don't know if we can have an accurate idea of this or if it varies too much between cases.

[Rekyt]

If we implement the suggestion from #80 with use_memoise() we could test if the plan is executed sequentially or in parallel, and if it's executed in parallel with the memoise option on show a message to the user that fundiversity fellback to the unmemoised version of functions.

We can check if the plan is sequential or not using plan() without any arguments like the following:

library("future")

# Sequential plan
plan(sequential)
inherits(plan(), "sequential")
#> [1] TRUE

# Other plans
plan(multicore)
inherits(plan(), "sequential")
#> [1] FALSE

plan(multisession)
inherits(plan(), "sequential")
#> [1] FALSE

^{Created on 2023-12-03 with reprex v2.0.2}