'attempt to apply non-function' error using python functions through reticulate within future_lapply

[charliekirkwood]

Hi thanks for the great package, I'm wondering if there is a way to expose python functions (imported into the main R session using reticulate) to the parallel workers of future.apply in order to avoid 'attempt to apply non-function' errors?

Reproducible examples below (borrowed from your own example on stack overflow) - the first one works, the second one doesn't despite working for standard lapply:

# set up the R session
library(future.apply)
plan(multisession)

library(progressr)
handlers(handler_progress(format="[:bar] :percent :eta :message"))

library(reticulate)
np <- import("numpy")

# working example using native R function
xs <- 1:100
with_progress({
  p <- progressor(along=xs)
  results <- future_lapply(xs, FUN=function(x) {
    p()  ## signal progress
    Sys.sleep(0.1)
    sqrt(x)
  }, future.chunk.size=1)
})

# non-working example using equivalent numpy function
xs <- 1:100
with_progress({
  p <- progressor(along=xs)
  results <- future_lapply(xs, FUN=function(x) {
    p()  ## signal progress
    Sys.sleep(0.1)
    np$sqrt(x)
  }, future.chunk.size=1)
})

[HenrikBengtsson]

This is explained in the 'Non-Exportable Objects' vignette (https://cran.r-project.org/web/packages/future/vignettes/future-4-non-exportable-objects.html). There's no solution to this.

Here's a minimal reproducible example:

library(reticulate)
library(future)
plan(multisession, workers = 2)

np <- import("numpy")
f <- future(np$sqrt(x))
value(f)
## Error in withCallingHandlers({ : attempt to apply non-function

[charliekirkwood]

Ok thanks for the clarification