progress bar

[talegari]

Hi Henrik,

Great package and work!

Is it feasible to include a progress for any future_* function?

Currently, I use pbmcapply for this purpose(limited in utility as compared to future though)

Regards, Srikanth KS

[xiaodaigh]

FWIW, the furrr package provides progress bars using the .progress argument.

[HenrikBengtsson]

The progressr package was created for the purpose of being able to get progress updates in any context, including when doing parallel processing. It's makes no assumptions about the future framework, and vice verse.

Here's an example from its README:

library(future.apply)
plan(multisession)

library(progressr)
handlers("progress", "beepr")

xs <- 1:5

with_progress({
  p <- progressor(along = xs)
  y <- future_lapply(xs, function(x, ...) {
    p(sprintf("x=%g", x))
    Sys.sleep(6.0-x)
    sqrt(x)
  })
})

[HenrikBengtsson]

UPDATE: With future 1.16.0 - on CRAN since 2020-01-16 - the example in https://github.com/HenrikBengtsson/future.apply/issues/34#issuecomment-549011124 will report on progress updates in a near-live fashion.

[mem48]

Hi are there any more example of how to do this:

I'm following the example in the progressr vignette:

library(future.apply)
plan(multisession)

library(progressr)
handlers(global = TRUE)
handlers("progress", "beepr")

my_fcn <- function(xs) {
  p <- progressor(along = xs)
  y <- future_lapply(xs, function(x, ...) {
    Sys.sleep(6.0-x)
    p(sprintf("x=%g", x))
    sqrt(x)
  })
}

my_fcn(1:5)

But in my case, I want to call a named function rather than include the code within the future_lapply

my_fcn <- function(xs) {
  p <- progressor(along = xs)
  y <- future_lapply(xs, inner_func)
}

inner_func <- function(x){
  Sys.sleep(6.0-x)
  p(sprintf("x=%g", x))
  sqrt(x)
}

my_fcn(1:5)

I get the error

Error in p(sprintf("x=%g", x)) : could not find function "p" 

I also tried adding the ... but then I get errors

Error in my_fcn(1:5) : '...' used in an incorrect context

Any suggestions on how to fix this?

[HenrikBengtsson]

Hi. So just to be clear, this is how R works, i.e. it's unrelated to the future framework and the progressr package per se.

Here's an example without either:

my_fcn <- function(xs) {
  p <- function(...) message(...)
  y <- lapply(xs, inner_func)
}

inner_func <- function(x) {
  p(sprintf("x=%g", x))
  sqrt(x)
}

which gives:

> my_fcn(1:3)
Error in p(sprintf("x=%g", x)) : could not find function "p"

One way to solve this is to pass p as an argument to inner_func(), e.g.

my_fcn <- function(xs) {
  p <- function(...) message(...)
  y <- lapply(xs, inner_func, p = p)
}

inner_func <- function(x, p) {
  p(sprintf("x=%g", x))
  sqrt(x)
}

which works, e.g.

> my_fcn(1:3)
x=1
x=2
x=3

To clarify further, the name p is not used, so it'll also work if you do:

my_fcn <- function(xs) {
  my_p <- function(...) message(...)
  y <- lapply(xs, inner_func, p = my_p)
}

[mem48]

Thanks, I thought it would be something simple like that.