Closed Rekyt closed 6 months ago
Thanks for starting this discussion, this is helpful!
My intuition for R.cache
is that it's not worth it in our case. Our tasks are still relatively fast and I'm not sure we can compensate the overhead of opening & writing to file.
We can properly benchmark it at some point. I guess it will depend on how many times the memoised results are reused though and I don't know if we can have an accurate idea of this or if it varies too much between cases.
If we implement the suggestion from #80 with use_memoise()
we could test if the plan is executed sequentially or in parallel, and if it's executed in parallel with the memoise option on show a message to the user that fundiversity fellback to the unmemoised version of functions.
We can check if the plan is sequential or not using plan()
without any arguments like the following:
library("future")
# Sequential plan
plan(sequential)
inherits(plan(), "sequential")
#> [1] TRUE
# Other plans
plan(multicore)
inherits(plan(), "sequential")
#> [1] FALSE
plan(multisession)
inherits(plan(), "sequential")
#> [1] FALSE
Created on 2023-12-03 with reprex v2.0.2
We officially recommended not to use
memoise
andfuture
at the same time in thefundiversity
manuscript. However, there maybe ways to get both.I will collect here possibilities to work with both:
plumber
,future
, andmemoise
together: https://stackoverflow.com/q/70805314future
repo: https://github.com/HenrikBengtsson/future/discussions/506future
) suggesting to useR.cache
file caching system instead: https://stackoverflow.com/a/48102804memoise
repo: https://github.com/r-lib/memoise/issues/29It seems for now that using
memoise
it's not straightforward to parallelize. Maybe we should be extra-careful and add a warning when loading the package with memoisation that it shouldn't be used with parallelization. We should also add this in: