Poor Julia performance when benchmarking against C++ and R?

[wleoncio]

I'm writing a short script to compare the performance of Julia against C++ and R, using an R script as the main file reproduced below:

# Loading packages
library(Rcpp)
library(RcppArmadillo)
library(JuliaConnectoR)
library(microbenchmark)

# Generating data
set.seed(9392927)
N <- 1e3L
x1 <- matrix(rpois(N ^ 2L, lambda = 10L), N)
x2 <- matrix(rpois(N ^ 2L, lambda = 10L), N)

# Defining functions
mySolveR <- function(m) {
  return(solve(m))
}
myProdR <- function(m1, m2) {
  return(m1 %*% m2)
}
cppFunction("
  arma::mat mySolveCpp(arma::mat m) {
    return(inv(m));
  }",
  depends = "RcppArmadillo"
)
cppFunction("
  arma::mat myProdCpp(arma::mat m1, arma::mat m2) {
    return(m1 * m2);
  }",
  depends = "RcppArmadillo"
)
mySolveJulia <- juliaEval("
  function mySolveJulia(m)
    return(inv(m))
  end"
)
#> Starting Julia ...
myProdJulia <- juliaEval("
  function myProdJulia(m1, m2)
    return(m1 * m2)
  end"
)

# Benchmarking
reps <- 10L
print(
  microbenchmark(
      mySolveR(x1), mySolveCpp(x1), mySolveJulia(x1),
      times = reps
    )
)
#> Unit: milliseconds
#>              expr        min         lq      mean     median        uq
#>      mySolveR(x1)   41.53371   44.03872  186.4527   59.43063  465.0983
#>    mySolveCpp(x1)   49.42867   72.63049  240.0839   81.16933  440.1176
#>  mySolveJulia(x1) 3856.43147 3897.36013 5112.4448 5376.94133 6026.1369
#>        max neval cld
#>   528.8838    10  a 
#>   576.2336    10  a 
#>  6220.1093    10   b
print(
  microbenchmark(
    myProdR(x1, x2), myProdCpp(x1, x2), myProdJulia(x1, x2),
    times = reps
  )
)
#> Unit: milliseconds
#>                 expr        min         lq       mean     median         uq
#>      myProdR(x1, x2)   20.10827   23.50031   27.69398   25.18961   32.51614
#>    myProdCpp(x1, x2)   25.59671   29.59619   38.32624   31.83016   49.33401
#>  myProdJulia(x1, x2) 5698.10212 6135.62034 6353.71632 6412.39110 6487.12516
#>         max neval cld
#>    39.89406    10  a 
#>    57.82850    10  a 
#>  7243.81362    10   b

^{Created on 2022-12-14 with reprex v2.0.2}

I am wondering why Julia is performing so poorly. My only guesses had to do with unnecessary recompilation of code or of Julia instances but using startJuliaServer()/stopJulia() doesn't help, and neither does isolating the functions in their own source files. What am I missing?

[stefan-m-lenz]

For communicating with Julia, the JuliaConnectoR uses TCP. This requires IO operations which will add to the execution time. Standard operations like matrix multiplication or matrix inversion are already highly optimized in R and the differences to Julia will be only minimal. The IO operations is simply much slower than the execution of these highly optimized functions on memory. You won't get a benefit from Julia or the JuliaConnectoR in this case. You can only expect an advantage over R if you have larger portions of code in Julia that cannot be optimized that much in R.

[wleoncio]

Thank you for the explanation. My intention is to eventually replace the simple functions above with more complicated ones that simulate real usage in our workplace. Perhaps I should go straight to those, though I did hope to be able to show that all three languages performed at the same order of magnitude for the simpler example.

[stefan-m-lenz]

With the JuliaConnectoR, you have some more communication overhead for the IO via TCP. So you can't use it to measure the Julia performance itself via measuring the execution time of the whole operation in R.

[wleoncio]

I see. In any case, I guess the example in my OP is a good counterexample (for when using a second language is not worth the effort).

FWIW, I just implemented a more complex example using linear search (inspired by this post) and got the performance differences I was expecting. \o/

[stefan-m-lenz]

I see. In any case, I guess the example in my OP is a good counterexample (for when using a second language is not worth the effort).

Yes, exactly.

FWIW, I just implemented a more complex example using linear search (inspired by this post) and got the performance differences I was expecting. \o/

Great to hear! Thanks for the link to the post.