search

JungiinChoi / SpMix

Building r package for a semiparametric density estimation
0 stars 0 forks source link

fmlogcondens compile issue #1

Open JungiinChoi opened 2 years ago

JungiinChoi commented 2 years ago

fmlogcondens: vignettes/documentation.Rmd

Installation

Standard installation

The most convenient way to install the package is:

install.packages("fmlogcondens")

The drawback of this approach is that the installed binaries where compiled for generic processors, which exclude AVX optimizations. See the next section, on how to activate AVX (and speedup the code by a factor of ~5x).

Installation with AVX support (Linux and Mac OS only) {#AVX}

AVX is the successor to SSE and MMX, and denotes the ability of modern CPUs to do certain simple computations (addition, multiplication, ...) much faster due to specialized instructions. To activate AVX, we have to install the package from source. To do this, we first download the source code from CRAN: link (Package source: fmlogondens_x.x.x.tar.gz).

Then in your favorite text editor open the file ~/.R/Makevars (create it if it does not exist) and add the line

CFLAGS += -march=native

This activates processor specific optimizations for the compilation of C code. [NOTE: This flag will affect all future installations that are manually compiled from source. In general it is a good idea to keep this flag after installing this pacakge, since it may potentially lead to faster code for other future packages too.]

Finally, from within R, we run the command:

install.packages("path-to-package/fmlogcondens_1.0.0.tar.gz", type = "source")

Alternatively we can execute the following command from the shell

R CMD INSTALL path-to-package/fmlogcondens_1.0.0.tar.gz

You can check whether AVX was activated for the installation by calling the function compilationInfo() in R after loading the package. The desired output would be AVX vector extensions activated..

A note on OpenMP and MacOS

OpenMP (Open Multi-Processing) is a API for C that supports multi-threaded programming. fmlogcondens automatically supports OpenMP. Problems can arise for MacOS when compiling from source, since the clang compiler may not support OpenMP. In that case you can try to follow this tutorial for a guide how to setup OpenMP under MacOS.

OpenMP

With the release of R 3.4.0, the R binary for macOS is compiled using a compiler and gfortran binaries that are not included with Xcode. However, unlike previous versions of R, these tools do provide support for OpenMP.

OFFICIAL GFORTRAN BINARY DOWNLOAD

Before we get too involved in the clang compiler specifics, you will need to download the official gfortran 6.1 build. Downloading and installing gfortran 6.1 allows for Fortran routines to benefit from OpenMP code. The official website where the installer is located is here: https://gcc.gnu.org/wiki/GFortranBinaries#MacOS-11

Note: You will need to download the OS X El Capitan gfortran 6.1 binaries regardless of whether or not you are on macOS Sierra, which presently only offers gfortran 6.3.

Clang

Verify the following statements are in the ~/.R/Makevars file:

CC=/usr/local/clang4/bin/clang
CXX=/usr/local/clang4/bin/clang++
CXX11=$CXX
CXX14=$CXX
CXX17=$CXX
CXX1X=$CXX
LDFLAGS=-L/usr/local/clang4/lib
JungiinChoi commented 2 years ago

Installation Error

ld: warning: directory not found for option '-L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin18/8.2.0'
ld: warning: directory not found for option '-L/usr/local/gfortran/lib'
ld: library not found for -lgfortran
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [SpMix.so] Error 1
ERROR: compilation failed for package ‘SpMix’
* removing ‘/Library/Frameworks/R.framework/Versions/4.1/Resources/library/SpMix’
* restoring previous ‘/Library/Frameworks/R.framework/Versions/4.1/Resources/library/SpMix’
Warning message:
In i.p(...) :
  installation of package ‘/var/folders/ms/1b6mj3f10_gbwfdztvcjv4b00000gn/T//RtmpKixSFq/file225b6226c088/SpMix_1.0.2.tar.gz’ had non-zero exit status

1. Downgrading gfortran

ld: warning: directory not found for option '-L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin18/8.2.0'
installing to /Library/Frameworks/R.framework/Versions/4.1/Resources/library/00LOCK-SpMix/00new/SpMix/libs
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (SpMix)
library(fmlogcondens)
library(LogConcDEAD)
set.seed(222)
X = matrix(rnorm(2000), 1000, 2)
# time estimate for our approach
system.time(r <- fmlcd(X))
#    user  system elapsed 
  1.130   0.012   1.148 
# time estimate for the approach of Cule et al.
system.time(rCule <- mlelcd(X))
#    user  system elapsed 
 64.749   0.391  65.302

2. Change Compiler to gcc

installing to /cloud/lib/x86_64-pc-linux-gnu-library/4.1/00LOCK-SpMix/00new/SpMix/libs
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (SpMix)
library(fmlogcondens)
library(LogConcDEAD)
set.seed(222)
X = matrix(rnorm(2000), 1000, 2)
# time estimate for our approach
system.time(r <- fmlcd(X))
   user  system elapsed 
 21.529   0.004  21.509 
# time estimate for the approach of Cule et al.
system.time(rCule <- mlelcd(X))
   user  system elapsed 
 70.970   0.138  71.238

3. Linux Server

library(fmlogcondens)
library(LogConcDEAD)
set.seed(222)
X = matrix(rnorm(2000), 1000, 2)
# time estimate for our approach
system.time(r <- fmlcd(X))
    user   system  elapsed 
1313.398    1.154  120.855 
# time estimate for the approach of Cule et al.
system.time(rCule <- mlelcd(X))
   user  system elapsed 
 89.943   0.197  90.007
JungiinChoi commented 2 years ago

Time Comparison

1. clang compile, gfortran version 6.3

image

2. gcc compile, gfortran version 11

image
JungiinChoi commented 2 years ago

4. Without Downgrading

library(fmlogcondens)
library(LogConcDEAD)
set.seed(222)
X = matrix(rnorm(2000), 1000, 2)
# time estimate for our approach
system.time(r <- fmlcd(X))
   user  system elapsed 
  5.881   5.848   3.547 
# time estimate for the approach of Cule et al.
system.time(rCule <- mlelcd(X))
   user  system elapsed 
 68.773   0.332  69.323

5. WHY??

Downloading GitHub repo JaneeChoi/SpMix@fmlogcondens_merge
✓  checking for file ‘/private/var/folders/s6/59y_sczx1yv3rt7vkp23_k1r0000gn/T/RtmpGukyiF/remotes45954bcae5b6/JaneeChoi-SpMix-c0af435/DESCRIPTION’ ...
─  preparing ‘SpMix’:
✓  checking DESCRIPTION meta-information ...
─  cleaning src
✓  checking vignette meta-information ...
─  checking for LF line-endings in source and make files and shell scripts
─  checking for empty or unneeded directories
   Omitted ‘LazyData’ from DESCRIPTION
─  building ‘SpMix_1.0.2.tar.gz’

Installing package into ‘/Users/choiiiiii/Library/R/x86_64/4.1/library’
(as ‘lib’ is unspecified)
* installing *source* package ‘SpMix’ ...
** using staged installation
** libs
clang -mmacosx-version-min=10.13 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../src  -I/usr/local/include   -fPIC  -Wall -g -O2  -c AVXinfo.c -o AVXinfo.o
clang -mmacosx-version-min=10.13 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../src  -I/usr/local/include   -fPIC  -Wall -g -O2  -c CNS.c -o CNS.o
clang -mmacosx-version-min=10.13 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../src  -I/usr/local/include   -fPIC  -Wall -g -O2  -c bfgsFullC.c -o bfgsFullC.o
clang -mmacosx-version-min=10.13 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../src  -I/usr/local/include   -fPIC  -Wall -g -O2  -c bfgsInitC.c -o bfgsInitC.o
clang -mmacosx-version-min=10.13 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../src  -I/usr/local/include   -fPIC  -Wall -g -O2  -c calcExactIntegral.c -o calcExactIntegral.o
clang -mmacosx-version-min=10.13 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../src  -I/usr/local/include   -fPIC  -Wall -g -O2  -c calcGradAVX.c -o calcGradAVX.o
clang -mmacosx-version-min=10.13 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../src  -I/usr/local/include   -fPIC  -Wall -g -O2  -c calcGradC.c -o calcGradC.o
clang -mmacosx-version-min=10.13 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../src  -I/usr/local/include   -fPIC  -Wall -g -O2  -c calcGradFastC.c -o calcGradFastC.o
clang -mmacosx-version-min=10.13 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../src  -I/usr/local/include   -fPIC  -Wall -g -O2  -c calcGradFastFloatC.c -o calcGradFastFloatC.o
clang -mmacosx-version-min=10.13 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../src  -I/usr/local/include   -fPIC  -Wall -g -O2  -c calcGradFloatC.c -o calcGradFloatC.o
clang -mmacosx-version-min=10.13 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../src  -I/usr/local/include   -fPIC  -Wall -g -O2  -c calcGradFullAVXC.c -o calcGradFullAVXC.o
clang -mmacosx-version-min=10.13 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../src  -I/usr/local/include   -fPIC  -Wall -g -O2  -c fmlogcondens_init.c -o fmlogcondens_init.o
clang -mmacosx-version-min=10.13 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../src  -I/usr/local/include   -fPIC  -Wall -g -O2  -c kernelDensC.c -o kernelDensC.o
clang -mmacosx-version-min=10.13 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../src  -I/usr/local/include   -fPIC  -Wall -g -O2  -c makeGridC.c -o makeGridC.o
clang -mmacosx-version-min=10.13 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../src  -I/usr/local/include   -fPIC  -Wall -g -O2  -c preCondGradAVXC.c -o preCondGradAVXC.o
clang -mmacosx-version-min=10.13 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../src  -I/usr/local/include   -fPIC  -Wall -g -O2  -c preCondGradFloatC.c -o preCondGradFloatC.o
clang -mmacosx-version-min=10.13 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../src  -I/usr/local/include   -fPIC  -Wall -g -O2  -c util.c -o util.o
clang -mmacosx-version-min=10.13 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/usr/local/lib -o SpMix.so AVXinfo.o CNS.o bfgsFullC.o bfgsInitC.o calcExactIntegral.o calcGradAVX.o calcGradC.o calcGradFastC.o calcGradFastFloatC.o calcGradFloatC.o calcGradFullAVXC.o fmlogcondens_init.o kernelDensC.o makeGridC.o preCondGradAVXC.o preCondGradFloatC.o util.o -L/Library/Frameworks/R.framework/Resources/lib -lRlapack -L/Library/Frameworks/R.framework/Resources/lib -lRblas -L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin18/8.2.0 -L/usr/local/gfortra
ld: warning: directory not found for option '-L/usr/local/gfortran/lib/gcc/x86_64-apple-darwin18/8.2.0'
ld: warning: directory not found for option '-L/usr/local/gfortran/lib'
ld: warning: dylib (/usr/local/lib/libgfortran.dylib) was built for newer macOS version (10.16) than being linked (10.13)
ld: warning: dylib (/usr/local/lib/libquadmath.dylib) was built for newer macOS version (10.16) than being linked (10.13)
installing to /Users/choiiiiii/Library/R/x86_64/4.1/library/00LOCK-SpMix/00new/SpMix/libs
** R
** inst
** byte-compile and prepare package for lazy loading
Warning: replacing previous import ‘mclust::dmvnorm’ by ‘mvtnorm::dmvnorm’ when loading ‘SpMix’
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
Warning: replacing previous import ‘mclust::dmvnorm’ by ‘mvtnorm::dmvnorm’ when loading ‘SpMix’
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
Warning: replacing previous import ‘mclust::dmvnorm’ by ‘mvtnorm::dmvnorm’ when loading ‘SpMix’
** testing if installed package keeps a record of temporary installation path
* DONE (SpMix)
> compilationInfo()
No vector extensions activated. (no AVX)
No parallel computing activated (no OMP).
choejeonuiiMac2:~ choiiiiii$ R CMD config CC
clang -mmacosx-version-min=10.13
choejeonuiiMac2:~ choiiiiii$ R CMD config FC
gfortran -mmacosx-version-min=10.13
choejeonuiiMac2:bin choiiiiii$ gfortran --version
bash: /usr/local/bin/gfortran: No such file or directory