Parallel option not working

[davidhen]

Hi Colin,

I was having a play around with this this morning. The standard function seems to work fine, but when I use the cores = .. argument I get an error as below. I've tried with 2, 4, and 8 cores with the same result.

# Programming benchmarks (5 tests):
# starting worker pid=10496 on localhost:11728 at 09:56:33.005
# starting worker pid=10495 on localhost:11728 at 09:56:33.005
# Error in unserialize(node$con) : error reading from connection
# Calls: <Anonymous> -> slaveLoop -> makeSOCKmaster
# Error in unserialize(node$con) : error reading from connection
# Calls: <Anonymous> -> slaveLoop -> makeSOCKmaster
# Execution halted
# Execution halted
# Error in makePSOCKcluster(names = spec, ...) : 
#  Cluster setup failed. 2 of 2 workers failed to connect.

What is really weird, is that when I created a reprex for this issue, it seemed to work...

library(benchmarkme)

benchmark_std(cores = 2L)
#> # Programming benchmarks (5 tests):
#> # Matrix calculation benchmarks (5 tests):
#> # Matrix function benchmarks (5 tests):
#>      user system elapsed          test test_group cores
#> 2   0.020  0.001   0.546           fib       prog     2
#> 3   0.004  0.001   0.383           fib       prog     2
#> 4   0.003  0.000   0.377           fib       prog     2
#> 21  0.003  0.000   1.172           gcd       prog     2
#> 31  0.003  0.000   0.831           gcd       prog     2
#> 41  0.003  0.001   0.905           gcd       prog     2
#> 22  0.004  0.001   0.694       hilbert       prog     2
#> 32  0.004  0.000   0.306       hilbert       prog     2
#> 42  0.004  0.000   0.402       hilbert       prog     2
#> 23  0.003  0.000   1.245      toeplitz       prog     2
#> 33  0.003  0.000   1.211      toeplitz       prog     2
#> 43  0.003  0.000   1.211      toeplitz       prog     2
#> 24  0.004  0.000   1.151     escoufier       prog     2
#> 34  0.004  0.000   1.098     escoufier       prog     2
#> 44  0.003  0.000   1.089     escoufier       prog     2
#> 25  0.003  0.000   0.990         manip matrix_cal     2
#> 35  0.003  0.000   0.725         manip matrix_cal     2
#> 45  0.004  0.000   0.714         manip matrix_cal     2
#> 211 0.003  0.000   0.956         power matrix_cal     2
#> 311 0.003  0.000   0.792         power matrix_cal     2
#> 411 0.003  0.000   0.715         power matrix_cal     2
#> 221 0.004  0.000   1.437          sort matrix_cal     2
#> 321 0.003  0.001   1.200          sort matrix_cal     2
#> 421 0.004  0.000   1.172          sort matrix_cal     2
#> 231 0.003  0.000   9.929 cross_product matrix_cal     2
#> 331 0.003  0.000   9.361 cross_product matrix_cal     2
#> 431 0.003  0.000   9.455 cross_product matrix_cal     2
#> 241 0.003  0.000   1.126            lm matrix_cal     2
#> 341 0.003  0.000   1.112            lm matrix_cal     2
#> 441 0.004  0.000   1.088            lm matrix_cal     2
#> 26  0.003  0.000  20.535      cholesky matrix_fun     2
#> 36  0.004  0.000  20.175      cholesky matrix_fun     2
#> 46  0.003  0.001  20.170      cholesky matrix_fun     2
#> 212 0.004  0.001   3.566   determinant matrix_fun     2
#> 312 0.003  0.000   3.493   determinant matrix_fun     2
#> 412 0.003  0.000   3.476   determinant matrix_fun     2
#> 222 0.003  0.000   0.871         eigen matrix_fun     2
#> 322 0.003  0.000   0.841         eigen matrix_fun     2
#> 422 0.004  0.000   0.821         eigen matrix_fun     2
#> 232 0.003  0.000   0.619           fft matrix_fun     2
#> 332 0.004  0.001   0.595           fft matrix_fun     2
#> 432 0.003  0.000   0.578           fft matrix_fun     2
#> 242 0.003  0.000   3.026       inverse matrix_fun     2
#> 342 0.003  0.000   3.000       inverse matrix_fun     2
#> 442 0.003  0.000   2.923       inverse matrix_fun     2

^{Created on 2020-11-18 by the reprex package (v0.3.0.9001)}

Session info

``` r sessioninfo::session_info() #> ─ Session info ─────────────────────────────────────────────────────────────── #> setting value #> version R version 4.0.2 (2020-06-22) #> os macOS 10.16 #> system x86_64, darwin17.0 #> ui X11 #> language (EN) #> collate en_GB.UTF-8 #> ctype en_GB.UTF-8 #> tz Europe/London #> date 2020-11-18 #> #> ─ Packages ─────────────────────────────────────────────────────────────────── #> package * version date lib source #> assertthat 0.2.1 2019-03-21 [1] CRAN (R 4.0.0) #> benchmarkme * 1.0.4 2020-05-09 [1] CRAN (R 4.0.2) #> benchmarkmeData 1.0.4 2020-04-23 [1] CRAN (R 4.0.2) #> cli 2.1.0 2020-10-12 [1] CRAN (R 4.0.2) #> codetools 0.2-16 2018-12-24 [1] CRAN (R 4.0.2) #> crayon 1.3.4 2017-09-16 [1] CRAN (R 4.0.0) #> digest 0.6.27 2020-10-24 [1] CRAN (R 4.0.2) #> doParallel 1.0.16 2020-10-16 [1] CRAN (R 4.0.2) #> dplyr 1.0.2 2020-08-18 [1] CRAN (R 4.0.2) #> ellipsis 0.3.1 2020-05-15 [1] CRAN (R 4.0.0) #> evaluate 0.14 2019-05-28 [1] CRAN (R 4.0.0) #> fansi 0.4.1 2020-01-08 [1] CRAN (R 4.0.0) #> foreach 1.5.0 2020-03-30 [1] CRAN (R 4.0.2) #> fs 1.5.0 2020-07-31 [1] CRAN (R 4.0.2) #> generics 0.0.2 2018-11-29 [1] CRAN (R 4.0.0) #> glue 1.4.2 2020-08-27 [1] CRAN (R 4.0.2) #> highr 0.8 2019-03-20 [1] CRAN (R 4.0.0) #> htmltools 0.5.0 2020-06-16 [1] CRAN (R 4.0.0) #> httr 1.4.2 2020-07-20 [1] CRAN (R 4.0.0) #> iterators 1.0.12 2019-07-26 [1] CRAN (R 4.0.2) #> knitr 1.30 2020-09-22 [1] CRAN (R 4.0.2) #> lattice 0.20-41 2020-04-02 [1] CRAN (R 4.0.2) #> lifecycle 0.2.0 2020-03-06 [1] CRAN (R 4.0.0) #> magrittr 2.0.1 2020-11-17 [1] CRAN (R 4.0.2) #> Matrix 1.2-18 2019-11-27 [1] CRAN (R 4.0.2) #> pillar 1.4.6 2020-07-10 [1] CRAN (R 4.0.2) #> pkgconfig 2.0.3 2019-09-22 [1] CRAN (R 4.0.0) #> purrr 0.3.4 2020-04-17 [1] CRAN (R 4.0.0) #> R6 2.5.0 2020-10-28 [1] CRAN (R 4.0.2) #> reprex 0.3.0.9001 2020-11-18 [1] Github (tidyverse/reprex@d3fc4b8) #> rlang 0.4.8 2020-10-08 [1] CRAN (R 4.0.2) #> rmarkdown 2.5 2020-10-21 [1] CRAN (R 4.0.2) #> rstudioapi 0.11 2020-02-07 [1] CRAN (R 4.0.0) #> sessioninfo 1.1.1 2018-11-05 [1] CRAN (R 4.0.0) #> stringi 1.5.3 2020-09-09 [1] CRAN (R 4.0.2) #> stringr 1.4.0 2019-02-10 [1] CRAN (R 4.0.0) #> tibble 3.0.4 2020-10-12 [1] CRAN (R 4.0.2) #> tidyselect 1.1.0 2020-05-11 [1] CRAN (R 4.0.0) #> vctrs 0.3.4 2020-08-29 [1] CRAN (R 4.0.2) #> withr 2.3.0 2020-09-22 [1] CRAN (R 4.0.2) #> xfun 0.19 2020-10-30 [1] CRAN (R 4.0.2) #> yaml 2.2.1 2020-02-01 [1] CRAN (R 4.0.0) #> #> [1] /Library/Frameworks/R.framework/Versions/4.0/Resources/library ```

...but still doesn't work without doing a reprex. I'm guessing a dependency issue? Although I can't figure out what might be missing.

Apologies if this me being stupid... but I can't figure it out.

[csgillespie]

It's something to do with namespaces and parallel. I think if you explicitly load benchmarkme, you shouldn't get this issue. I'll investigate more when I get a minute

[csgillespie]

So I'm fairly sure this is caused when you don't call library(benchmarkme) first.

Unfortunately, I'm certain I don't know how to fix it and get the package on CRAN. In the parallel code, I'm calling non-exported functions, so adding a library(benchmarkme) call to clusterExport doesn't work.

I could export all private functions, but I really don't want to.

---

I've now added a vaguely helpful error message (in the next version). If anyone can think of a fix, feel free to comment.