ropensci / drake

An R-focused pipeline toolkit for reproducibility and high-performance computing
https://docs.ropensci.org/drake
GNU General Public License v3.0
1.34k stars 128 forks source link

Many rsessions mostly idle with `parallelism = "future"` #1369

Closed matthiasgomolka closed 3 years ago

matthiasgomolka commented 3 years ago

Prework

Before trying to create a reprex, I wanted to make sure there is no obvious short answer. If this is not the case, I'll try to create a reprex.

Description

I noticed that very often, many of the rsessions which should be used for parallel processing in my drake plan, are idle (see screenshot).

image

I'm sure that the general setup with future::plan() before drake::make is correct, because some targets really run in parallel as I expect.

Is there a possibility that this is related to setting format = "file" for dynamic targets? Whenever I noticed that only one session is actually active, drake was working on a dynamic target with format = "file".

Desired result

All dynamic subtargets should run in parallel.

Session info

```r > sessionInfo() R version 4.0.2 (2020-06-22) Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows 10 x64 (build 18363) Matrix products: default locale: [1] LC_COLLATE=German_Germany.1252 LC_CTYPE=German_Germany.1252 LC_MONETARY=German_Germany.1252 LC_NUMERIC=C [5] LC_TIME=German_Germany.1252 attached base packages: [1] stats graphics grDevices datasets utils methods base other attached packages: [1] knitr_1.30 arrow_4.0.0 lubridate_1.7.9 haven_2.4.0 readxl_1.3.1 fs_1.5.0 anytime_0.3.9 [8] stringi_1.5.3 purrr_0.3.4 data.table_1.14.0 drake_7.12.7 dobby_0.1.0.111 conflicted_1.0.4 loaded via a namespace (and not attached): [1] Rcpp_1.0.5 txtq_0.2.3 listenv_0.8.0 visNetwork_2.0.9 prettyunits_1.1.1 ps_1.4.0 [7] assertthat_0.2.1 digest_0.6.27 R6_2.5.0 cellranger_1.1.0 backports_1.2.0 evaluate_0.14 [13] pillar_1.4.7 rlang_0.4.10 settings_0.2.4 progress_1.2.2 rstudioapi_0.13 callr_3.5.1 [19] R.utils_2.10.1 R.oo_1.24.0 DT_0.16 checkmate_2.0.0 rmarkdown_2.5 qs_0.24.1 [25] stringr_1.4.0 htmlwidgets_1.5.2 igraph_1.2.6 bit_4.0.4 validate_0.9.3 compiler_4.0.2 [31] xfun_0.19 pkgconfig_2.0.3 askpass_1.1 globals_0.13.1 htmltools_0.5.0 openssl_1.4.3 [37] tidyselect_1.1.0 tibble_3.0.4 codetools_0.2-16 future_1.19.1 crayon_1.3.4 R.methodsS3_1.8.1 [43] jsonlite_1.7.1 lifecycle_0.2.0 magrittr_2.0.1 storr_1.2.4 RcppParallel_5.0.2 cli_2.5.0 [49] cachem_1.0.4 renv_0.13.2 ellipsis_0.3.1 filelock_1.0.2 vctrs_0.3.5 generics_0.1.0 [55] stringfish_0.15.1 fst_0.9.4 RApiSerialize_0.1.0 tools_4.0.2 forcats_0.5.0 bit64_4.0.5 [61] glue_1.4.2 hms_0.5.3 processx_3.4.4 parallel_4.0.2 fastmap_1.0.1 yaml_2.2.1 [67] base64url_1.4 memoise_2.0.0 ```
wlandau commented 3 years ago

make(parallelism = "future") deliberately waits for the checksums of files to agree before moving on to subsequent targets (to compensate for network file system-related delays, usually NFS on clusters). Otherwise, drake cannot be sure if downstream targets have the latest versions of upstream targets. Seems a likely explanation in your case.