RAM issue with progress bars running on multisession future_apply

[ARLundborg]

Consider running the following dummy code:

library(parallelly)
library(parallel)
library(progressr)
library(future)
library(future.apply)

plan(multisession, workers=2, setup_strategy="sequential") 

x <- 1:10000
with_progress({
  p <- progressor(along=x)
  res <- future_sapply(x, FUN = function(x) {
                            p()
                            1+1
                             })
})

When I do this on my Macbook (with sessionInfo as seen at the bottom of this post) I get insane levels of RAM usage (upwards of 5 Gb used towards the end of the apply-function, scaling with the amount of calls to the progress bar). This memory will not release unless I explicitly gc (sometimes several times) and if I cancel the running of the apply function halfway through, the memory cannot be retrieved unless I restart R. The problem only occurs when running the progress bar with workers >1. If workers = 1 then no RAM issue occurs.

Contrast this with the same operation without a progress bar:

res <- future_sapply(x, FUN = function(x) {
    1+1
  })

which runs without any RAM issues whatsoever. I've created this example based on an identical issue that I was having with a future_apply run on a remote cluster.

Am I abusing the progressr-package somehow or is there a memory problem somewhere?

Here is my sessionInfo:

R version 4.1.0 (2021-05-18)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Big Sur 11.6.1

Matrix products: default
LAPACK: /Library/Frameworks/R.framework/Versions/4.1/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] parallel  stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] future.apply_1.8.1 future_1.21.0      progressr_0.8.0    parallelly_1.26.0 

loaded via a namespace (and not attached):
[1] compiler_4.1.0   tools_4.1.0      listenv_0.8.0    codetools_0.2-18 digest_0.6.28    globals_0.14.0  

[HenrikBengtsson]

Most likely fixed in progressr 0.9.0, which is on CRAN since end of September, cf. https://progressr.futureverse.org/news/index.html

[HenrikBengtsson]

You're also quite behind for the other "future" packages too, so I recommend updating them all.

[ARLundborg]

Hi Henrik, thanks for the quick response, you're right, I'll be sure to update my packages a bit more often to avoid these blunders!