unable to build r packge

[beew]

Description

It seems that I can't really get any consistent documentation on how to build the r package for mxnet with gpu. I have already successfully built mxnet with this command and the python package

make -j8 USE_OPENCV=1 USE_BLAS=openblasl USE_CUDA=1 USE_CUDA_PATH=/usr/local/cuda USE_CUDNN=1 CUDA_ARCH=-gencode=arch=compute_61,code=sm_61 USE_JEMALLOC=1

But there is little info on what to do next to install (and make ) the r package (or anything else besides python for that matter) Almost everything I found is about installing the mxnet-r binary and the README sends me to this page which is supposed to tell you how to install from source only talks about python https://mxnet.apache.org/versions/1.6/get_started/build_from_source#install-the-mxnet-package-for-r

The closest I can find is this but it doesn't work https://mxnet.apache.org/versions/1.4.1/install/ubuntu_setup.html#install-the-mxnet-package-for-r

First of all it says

 make rpkg

but it doesn't tell you running it from where. It doesn't work in mxnet's root

Makefile:358: WARNING: Significant performance increases can be achieved by installing and enabling gperftools or jemalloc development packages
Makefile:692: warning: overriding recipe for target 'build/libcustomop_gpu_lib.so'
Makefile:680: warning: ignoring old recipe for target 'build/libcustomop_gpu_lib.so'
Makefile:695: warning: overriding recipe for target 'build/libsubgraph_lib.so'
Makefile:683: warning: ignoring old recipe for target 'build/libsubgraph_lib.so'
make: *** No rule to make target 'rpkg'.  Stop.

and it doesn't work if run in the R-package directory

make rpkg
mkdir -p R-package/inst/libs
cp src/io/image_recordio.h R-package/src
cp: cannot stat 'src/io/image_recordio.h': No such file or directory
Makefile:5: recipe for target 'rpkg' failed

So it appears that this makefile in R-package should be one level up. So I copy it to mxnet root and backed up the original make file there

make rpkg
mkdir -p R-package/inst/libs
cp src/io/image_recordio.h R-package/src
if [ -d "lib" ]; then \
    cp -rf lib/libmxnet.so R-package/inst/libs; \
    if [ -e "lib/libtvm_runtime.so" ]; then \
        cp -rf lib/libtvm_runtime.so R-package/inst/libs; \
    fi; \
else \
    cp -rf build/libmxnet.so R-package/inst/libs; \
    if [ -e "build/libtvm_runtime.so" ]; then \
        cp -rf build/libtvm_runtime.so R-package/inst/libs; \
    fi; \
fi
mkdir -p R-package/inst/include
cp -rl include/* R-package/inst/include
Rscript -e "if(!require(devtools)){install.packages('devtools', repo = 'https://cloud.r-project.org/')}"
Loading required package: devtools
Loading required package: usethis
Rscript -e "if(!require(roxygen2)||packageVersion('roxygen2') < '6.1.1'){install.packages('roxygen2', repo = 'https://cloud.r-project.org/')}"
Loading required package: roxygen2
Rscript -e "library(devtools); library(methods); options(repos=c(CRAN='https://cloud.r-project.org/')); install_deps(pkg='R-package', dependencies = TRUE)"
Loading required package: usethis
Error: Can't find 'R-package'.
Execution halted
Makefile:5: recipe for target 'rpkg' failed

Am I missing something?

R 4.0.2 Ubuntu 16.04

[szha]

cc @hetong007

[hetong007]

I did the following on my Ubuntu to install the CPU-version MXNet, could you please help to follow and verify:

First, clone MXNet by git clone --recursive https://github.com/apache/incubator-mxnet.git -b 1.6.0.

Then, Compile with cmake:

cd $MXNET_ROOT # go into your cloned repo
mkdir build && cd build
cmake -DUSE_CUDA=OFF ..
make -j16

I believe you have been here already. So the R part is the following.

I have to modify two files first, please see the diff info below:

diff --git a/Makefile b/Makefile
index 4746cc434..0ec177232 100644
--- a/Makefile
+++ b/Makefile
@@ -707,6 +707,7 @@ rpkg:
        R CMD INSTALL R-package
        Rscript -e "require(mxnet); mxnet:::mxnet.export('R-package'); warnings()"
        rm R-package/NAMESPACE
+       touch R-package/NAMESPACE
        Rscript -e "devtools::document('R-package');warnings()"
        R CMD INSTALL R-package

diff --git a/R-package/R/zzz.R b/R-package/R/zzz.R
index a4f03ace0..1a421f1d4 100644
--- a/R-package/R/zzz.R
+++ b/R-package/R/zzz.R
@@ -34,6 +34,7 @@ NULL
 .onLoad <- function(libname, pkgname) {
   # Require methods for older versions of R
   require(methods)
+  require(Rcpp)
   tryCatch(library.dynam("libmxnet", pkgname, libname, local=FALSE), error = function(e) { print('Loading local: inst/libs/libmxnet.so'); dyn.load("R-package/inst/libs/libmxnet.so", local=FALSE) })
   tryCatch(library.dynam("mxnet", pkgname, libname), error = function(e) { print('Loading local: src/mxnet.so'); dyn.load("R-package/src/mxnet.so") })
   loadModule("mxnet", TRUE)

Finally, under your $MXNET_ROOT, run make rpkg.

[beew]

@hetong007 Didn't work

make rpkg
Makefile:227: "USE_LAPACK disabled because libraries were not found"
Makefile:355: WARNING: Significant performance increases can be achieved by installing and enabling gperftools or jemalloc development packages
mkdir -p R-package/inst/libs
cp src/io/image_recordio.h R-package/src
cp -rf lib/libmxnet.so R-package/inst/libs
if [ -e "lib/libdnnl.so.1" ]; then \
    cp -rf lib/libdnnl.so.1 R-package/inst/libs; \
fi
if [ -e "lib/libtvm_runtime.so" ]; then \
    cp -rf lib/libtvm_runtime.so R-package/inst/libs; \
fi
mkdir -p R-package/inst/include
cp -rl include/* R-package/inst/include
Rscript -e "if(!require(devtools)){install.packages('devtools', repo = 'https://cloud.r-project.org/')}"
Loading required package: devtools
Loading required package: usethis
Rscript -e "if(!require(roxygen2)||packageVersion('roxygen2') < '6.1.1'){install.packages('roxygen2', repo = 'https://cloud.r-project.org/')}"
Loading required package: roxygen2
Rscript -e "library(devtools); library(methods); options(repos=c(CRAN='https://cloud.r-project.org/')); install_deps(pkg='R-package', dependencies = TRUE)"
Loading required package: usethis
Error: Can't find 'R-package'.
Execution halted
make: *** [Makefile:697: rpkg] Error 1

my makefile (in mxnet root) now looks like this

R CMD INSTALL R-package
Rscript -e "require(mxnet); mxnet:::mxnet.export('R-package'); warnings()"
touch R-package/NAMESPACE
Rscript -e "devtools::document('R-package');warnings()"
R CMD INSTALL R-package

Also the warnings look like it is trying to build mxnet from scratch, I have already built successfully with

make -j4 USE_OPENCV=1 USE_BLAS=openblas USE_CUDA=0 USE_CUDNN=0 USE_MKLDNN=1 USE_JEMALLOC=1

So why is it still saying "WARNING: Significant performance increases can be achieved by installing and enabling gperftools or jemalloc development packages"?

[hetong007]

Make sure you cloned the branch 1.6.0 or v1.x. I believe the current master branch is undergoing some major change and the folder R-package is currently removed.

To clone a specific branch, use

git clone --recursive https://github.com/apache/incubator-mxnet.git -b 1.6.0

or

git clone --recursive https://github.com/apache/incubator-mxnet.git -b v1.x

[beew]

It is branch 1.6.0

python
Python 3.7.9 (default, Aug 24 2020, 16:50:14) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import mxnet
>>> mxnet.__version__
'1.6.0'
>>> 

[hetong007]

Your error message looks like that the path R-package doesn't exist, and which shouldn't be. That might because you have cloned a different version, or your folder has been modified unexpectedly.

1. Could you show me the output of git status under $MXNET_ROOT
2. Instead of make rpkg, could you manually execute these commands in your $MXNET_ROOT line-by-line in your console?

[beew]

Hi

git status
On branch v1.6
Your branch is up to date with 'origin/v1.6.x'.

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
    modified:   Makefile
    modified:   R-package/R/zzz.R

Untracked files:
  (use "git add <file>..." to include in what will be committed)
    Makefile.orig
    Makefile.rej
    R-package/R/zzz.R.orig
    R-package/R/zzz.R.rej
    R-package/src/image_recordio.h

no changes added to commit (use "git add" and/or "git commit -a")

I can indeed see R-packge

:~/opt/mxnet$ ls | grep R-package
R-package

[hetong007]

Have you tried to execute the commands line-by-line here: https://github.com/apache/incubator-mxnet/blob/1.6.0/Makefile#L687-L711 ?

According to your error message, it breaks at install_deps(pkg='R-package', dependencies = TRUE), and by manual execution we could simulate the environment right before this line and we debug from there.

[beew]

It breaks here

Rscript -e "library(devtools); library(methods); options(repos=c(CRAN='https://cloud.r-project.org/')); install_deps(pkg='R-package', dependencies = TRUE)"
Loading required package: usethis
Error: Can't find 'R-package'.
Execution halted

[hetong007]

If you start R and execute them in the console, do you still get the same error?

[beew]

Execute what in R? Those aren't R codes.

[hetong007]

These are R codes:

library(devtools)
library(methods)
options(repos=c(CRAN='https://cloud.r-project.org/'))
install_deps(pkg='R-package', dependencies = TRUE)

[beew]

The last line produces

Error: Can't find 'R-package'.

[hetong007]

In the same R console, what are the outputs from these commands:

sessionInfo()
list.files()
getwd()

[beew]

> sessionInfo()
R version 4.0.2 (2020-06-22)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.04.1 LTS

Matrix products: default
BLAS/LAPACK: /usr/lib/x86_64-linux-gnu/libmkl_rt.so

locale:
 [1] LC_CTYPE=en_CA.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_CA.UTF-8        LC_COLLATE=en_CA.UTF-8    
 [5] LC_MONETARY=en_CA.UTF-8    LC_MESSAGES=en_CA.UTF-8   
 [7] LC_PAPER=en_CA.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_CA.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] devtools_2.3.1 usethis_1.6.1 

loaded via a namespace (and not attached):
 [1] prettyunits_1.1.1 ps_1.3.4          fansi_0.4.1       rprojroot_1.3-2  
 [5] withr_2.2.0       digest_0.6.25     crayon_1.3.4      assertthat_0.2.1 
 [9] R6_2.4.1          backports_1.1.9   magrittr_1.5      rlang_0.4.7      
[13] cli_2.0.2         remotes_2.2.0     fs_1.5.0          testthat_2.3.2   
[17] callr_3.4.3       ellipsis_0.3.1    desc_1.2.0        tools_4.0.2      
[21] glue_1.4.2        pkgload_1.1.0     compiler_4.0.2    processx_3.4.3   
[25] pkgbuild_1.1.0    sessioninfo_1.1.1 memoise_1.1.0    
> list.files()
 [1] "acas"                         "cairoR2.R"                   
 [3] "cairotest.R"                  "CMHAcomplete-v27.sav"        
 [5] "combinelmeroutputs.R"         "csdtest.R"                   
 [7] "csdwebtest.R"                 "experiments"                 
 [9] "Feast"                        "funclib.R"                   
[11] "graphdata.R"                  "gridtest.R"                  
[13] "growthcurve"                  "imortdata.R"                 
[15] "importSPSS.R"                 "lib"                         
[17] "Lin"                          "LMM_Examples.R"              
[19] "machine_learning"             "MakeVars.R"                  
[21] "MathemaTIC"                   "medical"                     
[23] "medical_record"               "mentalabacus-master.zip"     
[25] "ML"                           "multi"                       
[27] "nvblas.log"                   "OUTPUTS"                     
[29] "plumlytics"                   "pqR"                         
[31] "RAnalyticFlow"                "random_notes_and_exercises.R"
[33] "rooksource.R~"                "RookTutorial"                
[35] "RookTutorial.R"               "simulation_book.R"           
[37] "speech_recognition"           "tempcsv.R"                   
[39] "test.R"                       "testdata"                    
[41] "testout"                      "textmining-exercises"        
[43] "time.to.event.sample.R"       "to do"                       
[45] "Venndiagrams.R"               "winbugs_tests"               
[47] "x86_64-pc-linux-gnu-library"  "YeeHong"                     
> getwd()
[1] "/home/bee/R"

Ok I setwd('/home/bee/R') in .Rprofile is that an issue since some packages assumes it is $HOME?

[hetong007]

From these outputs, your R is not started under $MXNET_ROOT, then no doubt that install_deps(pkg='R-package', dependencies = TRUE) can't find the folder R-package.

Would you start R in $MXNET_ROOT and run the following again:

sessionInfo()
list.files()
getwd()

[hetong007]

Ok I setwd('/home/bee/R') in .Rprofile is that an issue since some packages assumes it is $HOME?

Just saw this line. Yes, that is very likely to be the cause. Please remove this line and try make rpkg.

[beew]

It took a while, the R-package not found issue is solved but I got another error

g++ -std=gnu++11 -I"/usr/share/R/include" -DNDEBUG -I../inst/include -I'/usr/lib/R/site-library/Rcpp/include'    -fpic  -g -O2 -fdebug-prefix-map=/build/r-base-5iUtQS/r-base-4.0.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c export.cc -o export.o
g++ -std=gnu++11 -I"/usr/share/R/include" -DNDEBUG -I../inst/include -I'/usr/lib/R/site-library/Rcpp/include'    -fpic  -g -O2 -fdebug-prefix-map=/build/r-base-5iUtQS/r-base-4.0.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -g  -c im2rec.cc -o im2rec.o
im2rec.cc:39:10: fatal error: opencv2/opencv.hpp: No such file or directory
   39 | #include <opencv2/opencv.hpp>

but I have already installed libopencv-dev and opencv2/opencv.hpp is in /usr/include/opencv4/opencv2/opencv.hpp

Should I just make a symlink of /usr/include/opencv4/opencv2/ to /usr/local/include/opencv2?

[beew]

So I fixed the above by export CPATH=usr/include/opencv4 But then

im2rec.cc:172:29: error: ‘CV_IMWRITE_PNG_COMPRESSION’ was not declared in this scope
  172 |     encode_params.push_back(CV_IMWRITE_PNG_COMPRESSION);
      |                             ^~~~~~~~~~~~~~~~~~~~~~~~~~
im2rec.cc:176:29: error: ‘CV_IMWRITE_JPEG_QUALITY’ was not declared in this scope
  176 |     encode_params.push_back(CV_IMWRITE_JPEG_QUALITY);

[szha]

@hetong007 is there a way to build the R package with the libmxnet.so in pip packages?

[leezu]

@beew note that https://github.com/apache/incubator-mxnet/commit/9f2e73f39b405ec24cab254076f964452382c224 contains fixes for the R package and OpenCV 4. I recommend you use MXNet 1.7 which contains this commit

[hetong007]

is there a way to build the R package with the libmxnet.so in pip packages?

In theory you can simply replace this line: https://github.com/apache/incubator-mxnet/blob/1.6.0/Makefile#L690. But seems 1.7.0 has major changes in Makefile so I'm not sure how to work with that now.

[szha]

is there a way to build the R package with the libmxnet.so in pip packages?

In theory you can simply replace this line: https://github.com/apache/incubator-mxnet/blob/1.6.0/Makefile#L690. But seems 1.7.0 has major changes in Makefile so I'm not sure how to work with that now.

You can find this section here: https://github.com/apache/incubator-mxnet/blob/7a84fe11dd7399d1c2bdd0fb679c46af69e48861/R-package/Makefile#L4-L17

[hetong007]

@szha But isn't it confusing to have the relative path like R-package/inst/libs in this file?

[szha]

@szha But isn't it confusing to have the relative path like R-package/inst/libs in this file?

I'm not sure what the expectation is with this Makefile. @leezu probably knows better.

[beew]

@beew note that 9f2e73f contains fixes for the R package and OpenCV 4. I recommend you use MXNet 1.7 which contains this commit

Actually the makefile in V1.7 doesn't even have rpkg

make rpkg
Makefile:230: "USE_LAPACK disabled because libraries were not found"
Makefile:358: WARNING: Significant performance increases can be achieved by installing and enabling gperftools or jemalloc development packages
Makefile:692: warning: overriding recipe for target 'build/libcustomop_gpu_lib.so'
Makefile:680: warning: ignoring old recipe for target 'build/libcustomop_gpu_lib.so'
Makefile:695: warning: overriding recipe for target 'build/libsubgraph_lib.so'
Makefile:683: warning: ignoring old recipe for target 'build/libsubgraph_lib.so'
make: *** No rule to make target 'rpkg'.  Stop.

[hetong007]

As mentioned above, the rpkg part lies in the R-package/Makefile. You can probably copy the file to $MXNET_ROOT and then make rkpg.

If that doesn't work, @leezu Would you mind to show how to make with this target?

[leezu]

@beew If you use 1.7, please follow the documentation as contained in https://github.com/apache/incubator-mxnet/commit/9f2e73f39b405ec24cab254076f964452382c224

[beew]

As mentioned above, the rpkg part lies in the R-package/Makefile. You can probably copy the file to $MXNET_ROOT and then make rkpg.

It finally works!

There is one small issue, it complains that opencv.pc not found because the file is called opencv4.pc in Ubuntu 20.04 (from libopencv-dev) The fix is to create a symlink

sudo ln -s /usr/lib/x86_64-linux-gnu/pkgconfig/opencv4.pc  /usr/lib/x86_64-linux-gnu/pkgconfig/opencv.pc

Thanks @hetong007 for the help!

I think there should be better documentation on how to compile the r module (and perhaps everything else besides python) Rightnow all the public sources appear to be useless.

Thanks again.