doesn't get log or progress bar at real time

[D2Joy]

(Please use https://github.com/HenrikBengtsson/future/discussions for Q&A)

Describe the bug I originally have the option .verbose = T and .option with progress bar using doSNOW and tried to migrate to doFuture. However, no matter using registerDoFuture() or %dofuture% methods, I only got the the log for the first loop and then nothing updated. It won't give the rest of the log until the very end of the parallel processing, and no progress bar at all.

Reproduce example I have code just like below: progress <- function(n) setTxtProgressBar(txtProgressBar(max = length(filepaths_pos), style = 3), n) merged_slice<-foreach (i = 1:length(filepaths_pos), .combine=rbind, .verbose = T, .options.snow= list(progress = progress))%dofuture%{...} I also tried with .option.future, and still got no progress bar.

Expected behavior I expect the log and the progress bar to update at the real time, just like it behaves with doSNOW.

Session information Please share your session information, e.g.

> sessionInfo()
R version 3.6.3 (2020-02-29)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

Random number generation:
 RNG:     Mersenne-Twister 
 Normal:  Inversion 
 Sample:  Rounding 

locale:
[1] LC_COLLATE=Chinese (Simplified)_China.936  LC_CTYPE=Chinese (Simplified)_China.936   
[3] LC_MONETARY=Chinese (Simplified)_China.936 LC_NUMERIC=C                              
[5] LC_TIME=Chinese (Simplified)_China.936    

attached base packages:
[1] graphics  grDevices datasets  utils     parallel  stats     methods   base     

other attached packages:
 [1] roperators_1.3.14 data.table_1.14.8 psych_1.9.12      forcats_0.3.0     stringr_1.5.0     dplyr_1.0.10     
 [7] purrr_0.3.5       readr_1.3.1       tidyr_1.1.4       tibble_3.1.8      tidyverse_1.2.1   doSNOW_1.0.18    
[13] snow_0.4-3        iterators_1.0.13  doFuture_1.0.0    future_1.33.0     foreach_1.5.2     Hmisc_3.17-4     
[19] ggplot2_3.3.5     Formula_1.2-5     survival_3.1-8    lattice_0.20-38   magrittr_1.5      lme4_1.1-26      
[25] Matrix_1.2-18     scales_1.1.0     

loaded via a namespace (and not attached):
 [1] nlme_3.1-144        pbkrtest_0.4-7      lubridate_1.7.10    RColorBrewer_1.1-3  httr_1.4.2          tools_3.6.3        
 [7] utf8_1.1.4          R6_2.5.0            rpart_4.1-15        mgcv_1.8-31         DBI_1.1.1           colorspace_1.4-1   
[13] nnet_7.3-12         withr_2.5.0         tidyselect_1.1.2    gridExtra_2.3       mnormt_2.0.2        emmeans_1.7.5      
[19] compiler_3.6.3      quantreg_5.54       cli_3.6.1           rvest_1.0.3         SparseM_1.81        xml2_1.3.2         
[25] mvtnorm_1.1-3       digest_0.6.33       foreign_0.8-75      minqa_1.2.4         pkgconfig_2.0.3     parallelly_1.36.0  
[31] rlang_1.0.6         readxl_1.3.1        rstudioapi_0.13     generics_0.1.0      jsonlite_1.6        car_2.1-6          
[37] acepack_1.4.1       Rcpp_1.0.11         munsell_0.5.0       fansi_1.0.4         lifecycle_1.0.3     stringi_1.7.12     
[43] multcomp_1.2-21     yaml_2.3.7          MASS_7.3-51.5       plyr_1.8.8          grid_3.6.3          listenv_0.9.0      
[49] crayon_1.4.1        haven_1.1.2         splines_3.6.3       hms_0.4.2           tmvnsim_1.0-2       pillar_1.8.1       
[55] boot_1.3-24         estimability_1.4.1  future.apply_1.11.0 reshape2_1.4.4      codetools_0.2-16    glue_1.6.2         
[61] latticeExtra_0.6-26 modelr_0.1.8        vctrs_0.4.2         nloptr_1.2.2.2      MatrixModels_0.4-1  cellranger_1.1.0   
[67] gtable_0.3.0        xtable_1.8-4        broom_0.4.5         cluster_2.1.0       globals_0.16.2      statmod_1.4.37  

[HenrikBengtsson]

Describe the bug I originally have the option .verbose = T and .option with progress bar using doSNOW and tried to migrate to doFuture. However, no matter using registerDoFuture() or %dofuture% methods, I only got the the log for the first loop and then nothing updated. It won't give the rest of the log until the very end of the parallel processing, and no progress bar at all.

This is expected and by design. It's not a bug and you're not doing anything wrong.

To better understand why, it's helpful to understand that the doSNOW package is unique in how it report on progress. AFAIK, no other do... package implement progress updates by themselves. One could argue that the doSNOW progress reporter is rather ad hoc and quite limited, e.g. it is not providing "real time" progress reporting - it only reports on progress as parallel workers complete their tasks.

One could also argue that the developers of foreach, who also developed legacy doSNOW, could have implemented such progress reporting directly in foreach(...) %dopar% ... itself, but they choose not to. So, I take it that they were not satisfied with this solution and when they moved away from doSNOW to doParallel etc., they decided not to bring it in.

Now, to report on progress with the Futureverse, the recommend way is to use the progressr package. Contrary to doSNOW, it also has the capacity to report progress in a near-live fashion and with greater granularity.

See https://progressr.futureverse.org/#foreach-with-dofuture for an example how to use progressr with doFuture. That examples used %dopar% (will update that in the next release), but it works the same with %dofuture%.

PS. See if you can upgrade to a modern version of R; R 3.6.3 is quite old, and you're missing out on lots of great new features, including some affecting progress reporting, but also stability of parallel workers.

[D2Joy]

Hi, HenrikBengtsson. Thanks for the detailed explanation.

I see. I will need to use progressr package for progress then. How about .verbose = T? The log is also not showing at runtime.

[HenrikBengtsson]

How about .verbose = T?

It's currently not supported by doFuture, but I can see how it could be useful, especially since all other doNNN packages support it to some degree. I've added #80 to the to-do list.

The log is also not showing at runtime.

I'm not sure if this is a separate question, or related to .verbose = TRUE.

[D2Joy]

Thanks for the planning. That is the same issue I referred.