Error with custom methods in UNIX/HPC environment

[stephenashton-dhsc]

Hi,

If I try and run the following code on a Windows laptop through RStudio, there are no issues:

methods::setGeneric(
  "my_custom_method",
  function(x) {
    standardGeneric("my_custom_method")
  }
)

methods::setMethod(
  "my_custom_method",
  methods::signature(),
  function(x) {
    y <- x^2
    return(y)
  }
)

aa <- furrr::future_map(
  1:100,
  my_custom_method
)

print(aa)

However, if I get an error if I send this across to a HPC/UNIX environment using the following batch file:

#!/bin/bash

## Specify SLURM settings
#SBATCH --job-name=test
#SBATCH --mem-per-cpu=32000
#SBATCH --nodes=1
#SBATCH --distribution=cyclic
#SBATCH --partition=standard-wp
#SBATCH --exclusive

## Specify files to print output/error messages
#SBATCH --output=./logs/test-%j.out
#SBATCH --error=./logs/test-%j.error

## Setup HPC modules
module purge
module load openmpi
module load gcc
module load gdal
module load proj/proj-4.9.3
module load geos
module load udunits
module load sqlite
module load cmake
module load R
module load slurm

## Setup global variables
export R_LIBS_USER=/data/users/hemt/R-library
export TZ="Europe/London"
export CPPFLAGS="-I/usr/include"
export CXXFLAGS="-I/usr/include"
export CFLAGS="-I/usr/include"

## Launch R script
Rscript "./test.R"

The error message I get is as follows:

Error in (function (classes, fdef, mtable)  : 
  unable to find an inherited method for function ‘my_custom_method’ for signature ‘"integer"’
Calls: <Anonymous> ... resolve.list -> signalConditionsASAP -> signalConditions
Execution halted

I assume this is to do with the custom method, but am not sure what is causing it - any help you could give would be great! :)

[DavisVaughan]

Does it work if you use future alone? i.e. something like this, but with your specific plan() added in

library(future)

methods::setGeneric(
  "my_custom_method",
  function(x) {
    standardGeneric("my_custom_method")
  }
)

methods::setMethod(
  "my_custom_method",
  methods::signature(),
  function(x) {
    y <- x^2
    return(y)
  }
)

value(future(my_custom_method(1)))

[DavisVaughan]

If that doesn't work either, it is probably a future or future.batchtools (assuming you are using that) issue. You could probably work around it in the meantime by putting the generic and the method into a package and then making sure that package is loaded on the HPC worker.

[stephenashton-dhsc]

No, it's still not working. I'll submit an issue to the future package.

Many thanks! :)

[DavisVaughan]

CC @HenrikBengtsson