Closed rrichmond closed 1 year ago
The issue you are running into here is that the worker operates in its own environment, which does not have data.table
loaded.
If you tell clustermq to load the data.table package, it works as expected:
library(data.table)
library(clustermq)
library(foreach)
library(doParallel)
nvals <- 10
tmpdat <- data.table(a=1:nvals,b=nvals:1)
clustermq::register_dopar_cmq(n_jobs=4, pkgs=c("data.table"))
out <- foreach(idx=1:nrow(tmpdat)) %dopar% {
myrow <- tmpdat[idx]
myrow$a*myrow$b
}
out
#> [[1]]
#> [1] 10
#>
#> [[2]]
#> [1] 18
#>
#> [[3]]
#> [1] 24
#>
#> ...
Ideally, the error would tell you that the data.table
package is missing instead of a column subset error.
I will leave this open until I can find out if there is an easy solution to this.
Whoops I should have caught that. Thanks! This is definitely a tough one since [
will fall back to data.frame
s function.
Hello,
I'm having issues using clustermq with data.table. Here's a simple example:
Which returns
If instead I use doParallel everything works fine:
Any suggestions on how to debug or what is happening here? Thanks!