Compiling on macOS 10.15.2 with OpenMP

George9000 commented 4 years ago

Use of the detailed guide here on building with OpenMP on macOS 10.15 (environment variables in ~/.zshrc and an extensive Makevars file) along with the use of the homebrew package manager to install llvm and libomp led to a seemingly successful compilation of data.table from source with OpenMP.

However, the package messages prior to compilation seemed to indicate that OpenMP was not supported. Yet the compilation seems to have produced an openmp-utils object file and a getDTthreads showed multithreading capabilities. See output below. Was this an erroneous message during pre-compilation checks? Or perhaps, the final compiled product does not have OpenMP after all? See output below.

> install.packages("data.table", type = "source",
+                  repos = "http://Rdatatable.github.io/data.table")
trying URL 'http://Rdatatable.github.io/data.table/src/contrib/data.table_1.12.9.tar.gz'
Content type 'application/gzip' length 5187823 bytes (4.9 MB)
==================================================
downloaded 4.9 MB

* installing *source* package ‘data.table’ ...
** using staged installation
zlib 1.2.11 is available ok
*** OpenMP not supported! data.table uses OpenMP to automatically
***   parallelize operations like sorting, grouping, file reading, etc.
*** For details on how to install the necessary toolchains on your OS see:
***   https://github.com/Rdatatable/data.table/wiki/Installation
*** Continuing installation without OpenMP support...
** libs
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c assign.c -o assign.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c between.c -o between.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c bmerge.c -o bmerge.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c chmatch.c -o chmatch.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c cj.c -o cj.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c coalesce.c -o coalesce.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c dogroups.c -o dogroups.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c fastmean.c -o fastmean.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c fcast.c -o fcast.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c fifelse.c -o fifelse.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c fmelt.c -o fmelt.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c forder.c -o forder.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c frank.c -o frank.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c fread.c -o fread.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c freadR.c -o freadR.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c froll.c -o froll.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c frollR.c -o frollR.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c frolladaptive.c -o frolladaptive.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c fsort.c -o fsort.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c fwrite.c -o fwrite.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c fwriteR.c -o fwriteR.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c gsumm.c -o gsumm.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c ijoin.c -o ijoin.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c init.c -o init.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c inrange.c -o inrange.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c nafill.c -o nafill.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c nqrecreateindices.c -o nqrecreateindices.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c openmp-utils.c -o openmp-utils.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c quickselect.c -o quickselect.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c rbindlist.c -o rbindlist.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c reorder.c -o reorder.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c shift.c -o shift.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c subset.c -o subset.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c transpose.c -o transpose.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c types.c -o types.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c uniqlist.c -o uniqlist.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c utils.c -o utils.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c vecseq.c -o vecseq.o
/usr/local/opt/llvm/bin/clang -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -isysroot "/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk" -isystem "/usr/local/opt/llvm/include" -isystem "/usr/local/opt/gettext/include" -isystem "/usr/local/opt/icu4c/include"  -fPIC  -O3 -mtune=native -pipe -fopenmp -c wrappers.c -o wrappers.o
/usr/local/opt/llvm/bin/clang -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib --sysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -L /usr/local/opt/llvm/lib -L /usr/local/opt/gettext/lib -L /usr/local/opt/icu4c/lib -o data.table.so assign.o between.o bmerge.o chmatch.o cj.o coalesce.o dogroups.o fastmean.o fcast.o fifelse.o fmelt.o forder.o frank.o fread.o freadR.o froll.o frollR.o frolladaptive.o fsort.o fwrite.o fwriteR.o gsumm.o ijoin.o init.o inrange.o nafill.o nqrecreateindices.o openmp-utils.o quickselect.o rbindlist.o reorder.o shift.o subset.o transpose.o types.o uniqlist.o utils.o vecseq.o wrappers.o -lz -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
mv data.table.so datatable.so
if [ "" != "Windows_NT" ] && [ `uname -s` = 'Darwin' ]; then install_name_tool -id datatable.so datatable.so; fi
installing to /Library/Frameworks/R.framework/Versions/3.6/Resources/library/00LOCK-data.table/00new/data.table/libs
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (data.table)

The downloaded source packages are in
    ‘/private/var/folders/yh/bbqjrr3n06j6r9x6w_7_4hnh0000gn/T/RtmpMzZcdl/downloaded_packages’
> library(data.table)
data.table 1.12.9 IN DEVELOPMENT built 2020-01-03 02:11:59 UTC; travis using 8 threads (see ?getDTthreads).  Latest news: r-datatable.com
> ?getDTthreads
> getDTthreads(verbose=TRUE)
  omp_get_num_procs()            16
  R_DATATABLE_NUM_PROCS_PERCENT  unset (default 50)
  R_DATATABLE_NUM_THREADS        unset
  omp_get_thread_limit()         2147483647
  omp_get_max_threads()          16
  OMP_THREAD_LIMIT               unset
  OMP_NUM_THREADS                unset
  RestoreAfterFork               true
  data.table is using 8 threads. See ?setDTthreads.
[1] 8

Session Info:

> sessionInfo()
R version 3.6.2 Patched (2019-12-28 r77627)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS Catalina 10.15.2

Matrix products: default
BLAS:   /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRblas.0.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] data.table_1.12.9

loaded via a namespace (and not attached):
[1] compiler_3.6.2 tools_3.6.2

George9000 commented 4 years ago

Makevars file referenced above:

# https://mirror.its.sfu.ca/mirror/CRAN/bin/macosx/tools/
# https://github.com/Rdatatable/data.table/wiki/Installation
# https://asieira.github.io/using-openmp-with-r-packages-in-os-x.html
# https://thecoatlessprofessor.com/programming/openmp-in-r-on-os-x/
# https://mirror.its.sfu.ca/mirror/CRAN/bin/macosx/
# https://mirror.its.sfu.ca/mirror/CRAN/bin/macosx/tools/
# https://luispuerto.net/blog/2018/01/12/install-r-100-homebrew-edition-with-openblas-openmp-my-version/#data-table-package
# https://www.kthohr.com/r-mac-source.html

XCBASE:=$(shell xcrun --show-sdk-path)
LLVMBASE:=$(shell brew --prefix llvm)
GCCBASE:=$(shell brew --prefix gcc)
GETTEXT:=$(shell brew --prefix gettext)
ICU4C:=$(shell brew --prefix icu4c)

CC=$(LLVMBASE)/bin/clang
CXX=$(LLVMBASE)/bin/clang++
CXX11=$(LLVMBASE)/bin/clang++
CXX14=$(LLVMBASE)/bin/clang++
CXX17=$(LLVMBASE)/bin/clang++
CXX1X=$(LLVMBASE)/bin/clang++
SHLIB_CXXLD=$(LLVMBASE)/bin/clang++

CXX1XSTD=-std=c++0x

CPPFLAGS=-isysroot "$(XCBASE)" -isystem "$(LLVMBASE)/include" -isystem "$(GETTEXT)/include" -isystem "$(ICU4C)/include"
CFLAGS=-O3 -mtune=native -pipe
CXXFLAGS=-O3 -mtune=native -pipe
LDFLAGS=--sysroot "$(XCBASE)" -L "$(LLVMBASE)/lib" -L "$(GETTEXT)/lib" -L "$(ICU4C)/lib"
PKG_CXXFLAGS=-std=c++11

FC=$(GCCBASE)/bin/gfortran
F77=$(GCCBASE)/bin/gfortran
FLIBS=-L$(GCCBASE)/lib/gcc/9/ -lm
MAKE=make -j8

SHLIB_OPENMP_CFLAGS=-fopenmp
SHLIB_OPENMP_CXXFLAGS=-fopenmp
SHLIB_OPENMP_FCFLAGS=-fopenmp
SHLIB_OPENMP_FFLAGS=-fopenmp

# https://github.com/HenrikBengtsson/affxparser/issues/10
ifeq ($(shell $(CXX11) -fopenmp -E -xc++ - 2>&1 >/dev/null && echo 'true'), true)
  CFLAGS+=$(SHLIB_OPENMP_CFLAGS)
  CXXFLAGS+=$(SHLIB_OPENMP_CXXFLAGS)
  FCFLAGS+=$(SHLIB_OPENMP_FCFLAGS)
  FFLAGS+=$(SHLIB_OPENMP_FFLAGS)
endif

Relevant portion from ~/.zshrc

# Ensure clang knows where to find the macOS SDK.
XCBASE=`xcrun --show-sdk-path`
export C_INCLUDE_PATH=$XCBASE/usr/include
export CPLUS_INCLUDE_PATH=$XCBASE/usr/include
export LIBRARY_PATH=$XCBASE/usr/lib

Long, but possibly relevant, list of homebrew installed packages:

aspell
autoconf
automake
bash
cairo
cmake
coreutils
dos2unix
exa
fd
fontconfig
freetype
fribidi
fzf
gawk
gcc
gdbm
gdk-pixbuf
gettext
giflib
git
glib
gmp
gnu-sed
gnu-tar
gnutls
graphite2
harfbuzz
htop
icu4c
ilmbase
imagemagick
isl
jpeg
libcroco
libde265
libevent
libffi
libgcrypt
libgpg-error
libheif
libidn2
libmpc
libomp
libpng
librsvg
libtasn1
libtermkey
libtiff
libtool
libunistring
libuv
libvterm
libyaml
little-cms2
llvm
lua
lua@5.1
luajit
luarocks
lzo
mpfr
msgpack
multimarkdown
ncurses
neovim
nettle
nspr
nss
oniguruma
openconnect
openexr
openjpeg
openssl@1.1
p11-kit
pandoc
pandoc-citeproc
pandoc-crossref
pango
pcre
pcre2
pdfgrep
perl
pixman
pkg-config
poppler
python
qt
readline
rename
ripgrep
ruby
shared-mime-info
shellcheck
sqlite
stoken
swig
texinfo
tmux
tree
unbound
unibilium
vim
webp
wget
x265
xz
z
zsh

jangorecki commented 4 years ago

Thanks for detailed report.

jangorecki commented 4 years ago

@George9000 Could you please check if there is any difference now? PR #4374 changed the way how we detect openmp, so your issue might have been fixed.

George9000 commented 4 years ago

Thanks so much for the work on this issue. When it builds, it no longer gives the "OpenMP not supported" warning. See below.

> sessionInfo()
R version 4.0.2 (2020-06-22)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Catalina 10.15.5

Matrix products: default
BLAS:   /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRblas.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.0/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] pacman_0.5.1

loaded via a namespace (and not attached):
[1] compiler_4.0.2

After removing remove.packages("data.table) in R, I'm now building in the shell after cloning the data.table repository -- or doing a git pull if the repository is already present.

R CMD build data.table
[success]
R CMD INSTALL data.table_1.12.9.tar.gz
* installing to library ‘/Library/Frameworks/R.framework/Versions/4.0/Resources/library’
* installing *source* package ‘data.table’ ...
** using staged installation
zlib 1.2.11 is available ok
OpenMP supported
** libs
/usr/local/opt/llvm/bin/clang -fopenmp -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG   -I/usr/local/opt/gettext/include -I/usr/local/opt/llvm/include -I/usr/local/opt/libomp/include -Xpreprocessor -fopenmp   -fPIC  -g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe -c assign.c -o assign.o
In file included from assign.c:1:
In file included from ./data.table.h:13:
In file included from ./myomp.h:2:
/usr/local/opt/libomp/include/omp.h:53:9: warning: ISO C restricts enumerator
      values to range of 'int' (2147483648 is too large) [-Wpedantic]
        omp_sched_monotonic = 0x80000000
        ^                     ~~~~~~~~~~
/usr/local/opt/libomp/include/omp.h:302:7: warning: ISO C restricts enumerator
      values to range of 'int' (18446744073709551615 is too large) [-Wpedantic]
      KMP_ALLOCATOR_MAX_HANDLE = UINTPTR_MAX
      ^                          ~~~~~~~~~~~
/usr/local/opt/libomp/include/omp.h:315:7: warning: ISO C restricts enumerator
      values to range of 'int' (18446744073709551615 is too large) [-Wpedantic]
      KMP_MEMSPACE_MAX_HANDLE = UINTPTR_MAX
      ^                         ~~~~~~~~~~~
/usr/local/opt/libomp/include/omp.h:343:39: warning: ISO C restricts enumerator
      values to range of 'int' (18446744073709551615 is too large) [-Wpedantic]
    typedef enum omp_event_handle_t { KMP_EVENT_MAX_HANDLE = UINTPTR_MAX } o...
                                      ^                      ~~~~~~~~~~~
4 warnings generated.
...
installing to /Library/Frameworks/R.framework/Versions/4.0/Resources/library/00LOCK-data.table/00new/data.table/libs
** R
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** checking absolute paths in shared objects and dynamic libraries
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (data.table)

Those warnings are repeated multiple times. However, the install finishes successfully.

Note: To forestall problems, I finally just installed the entire XCode application, instead of just the xcode-select command-line tools.

My current messy .R/Makevars file:

# if you downloaded llvm manually above, replace with your chosen NEW_PATH/clang
LLVM_LOC = /usr/local/opt/llvm

## for data.table compilation, comment OUT these 2 lines and then see directions below for toggling 5 lines 
## when finished, reverse the changes
# CC=$(LLVM_LOC)/bin/clang
# CXX=$(LLVM_LOC)/bin/clang++
#
# O3 should be faster than -O2 (default) level optimisation ..
CFLAGS=-g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe
CXXFLAGS=-g -O3 -Wall -pedantic -std=c++11 -mtune=native -pipe
LDFLAGS=-L/usr/local/opt/gettext/lib -L$(LLVM_LOC)/lib -Wl,-rpath,$(LLVM_LOC)/lib -L/usr/local/opt/libomp/lib -lomp

CPPFLAGS=-I/usr/local/opt/gettext/include -I$(LLVM_LOC)/include

### For data.table only: toggle comments on 5 lines ####
# uncomment the TWO lines below and then the THREE lines starting with CPPFLAGS
CC=$(LLVM_LOC)/bin/clang -fopenmp
CXX=$(LLVM_LOC)/bin/clang++ -fopenmp

# see https://github.com/Rdatatable/data.table/issues/4348
# also, the -I/Library/Developer ... section might need to be removed, since XCode was installed, not just CommandLineTools. I've done this. See below.
# see https://stackoverflow.com/questions/58628377/catalina-c-using-cmath-headers-yield-error-no-member-named-signbit-in-th
# CPPFLAGS=-I/usr/local/opt/gettext/include -I$(LLVM_LOC)/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/usr/local/opt/libomp/include -Xpreprocessor -fopenmp

CPPFLAGS=-I/usr/local/opt/gettext/include -I$(LLVM_LOC)/include -I/usr/local/opt/libomp/include -Xpreprocessor -fopenmp
SHLIB_OPENMP_CFLAGS   =
SHLIB_OPENMP_CXXFLAGS =

Additional environment:

% xcrun --show-sdk-path
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk

My earlier post showed a portion of my .zshrc Those lines are now commented out.

# Ensure clang knows where to find the macOS SDK.
# XCBASE=`xcrun --show-sdk-path`
# export C_INCLUDE_PATH=$XCBASE/usr/include
# export CPLUS_INCLUDE_PATH=$XCBASE/usr/include
# export LIBRARY_PATH=$XCBASE/usr/lib

# export CPATH=`xcrun --show-sdk-path`/usr/include

My conclusion: to compile from source without headaches, just install the entire Xcode.app. And the latest changes to data.table show no more errors re: OpenMP.

Thanks for all the work. I love data.table

ChrissCod commented 3 years ago

@George9000 just to confirm, I am wondering, shouldn't these SHLIB_OPENMP_CFLAGS = SHLIB_OPENMP_CXXFLAGS =

be assigned to -fopenmp?

George9000 commented 3 years ago

@ChrissCod Sorry for taking a while to answer.

From data.table's code: src/configure (starting at line 69)

# Test if we have a OPENMP compatible compiler
# Aside: ${SHLIB_OPENMP_CFLAGS} does not appear to be defined at this point according to Matt's testing on
# Linux, and R CMD config SHLIB_OPENMP_CFLAGS also returns 'no information for variable'. That's not
# inconsistent with R-exts$1.2.1.1, though, which states it's 'available for use in Makevars' (so not
# necessarily here in configure). Hence use -fopenmp directly for this detection step.
# printf not echo to pass checkbashisms w.r.t. to the \n

cat <<EOF > test-omp.c
#include <omp.h>
int main() {
  return omp_get_num_threads();
}
EOF

# First, try R CMD SHLIB to see if R can already compile
# things using OpenMP without any extra help from data.table
"${R_HOME}/bin/R" CMD SHLIB test-omp.c >/dev/null 2>&1 || R_NO_OPENMP=1

if [ "$R_NO_OPENMP" = "1" ]; then
  # Compilation failed -- try forcing -fopenmp instead.
  # TODO: doesn't R_NO_OPENMP need to be set to 0 before next line?
  ${CC} ${CFLAGS} -fopenmp test-omp.c || R_NO_OPENMP=1
  # TODO: and then nothing seems to be done with this outcome
else
  echo "R CMD SHLIB supports OpenMP without any extra hint"
fi

I tried your suggestion in my ~/.R/Makevars for data.table

# if you downloaded llvm manually above, replace with your chosen NEW_PATH/clang
LLVM_LOC = /usr/local/opt/llvm

## for data.table compilation, comment OUT 2 lines below; then uncomment 5 lines in block at the end. Reverse after finishing.
# CC=$(LLVM_LOC)/bin/clang
# CXX=$(LLVM_LOC)/bin/clang++

# O3 should be faster than -O2 (default) level optimisation ..
CFLAGS=-g -O3 -Wall -pedantic -std=gnu99 -mtune=native -pipe
CXXFLAGS=-g -O3 -Wall -pedantic -std=c++11 -mtune=native -pipe
LDFLAGS=-L/usr/local/opt/gettext/lib -L$(LLVM_LOC)/lib -Wl,-rpath,$(LLVM_LOC)/lib -L/usr/local/opt/libomp/lib -lomp

CPPFLAGS=-I/usr/local/opt/gettext/include -I$(LLVM_LOC)/include

### For data.table only: uncomment these 5 lines ####
CC=$(LLVM_LOC)/bin/clang -fopenmp
CXX=$(LLVM_LOC)/bin/clang++ -fopenmp
CPPFLAGS=-I/usr/local/opt/gettext/include -I$(LLVM_LOC)/include -I/usr/local/opt/libomp/include -Xpreprocessor -fopenmp
SHLIB_OPENMP_CFLAGS   = -fopenmp
SHLIB_OPENMP_CXXFLAGS = -fopenmp

data.table compiled without trouble. First few lines of output below...

% R CMD INSTALL data.table_1.13.7.tar.gz                                                                                                     15:09
[1] "/Users/George/Documents/rsrc"
* installing to library ‘/Library/Frameworks/R.framework/Versions/4.0/Resources/library’
* installing *source* package ‘data.table’ ...
** using staged installation
zlib 1.2.11 is available ok
R CMD SHLIB supports OpenMP without any extra hint

Based on this, I suspect the two lines of SHLIB macros can be removed.

ChrissCod commented 3 years ago

@George9000. Thank you for your reply. Actually in my case, I used the assignation of these two SHLIB lines to -fopenmp and the installation went fine. I just asked because in your post of July 1, these two lines were not assigned to anything.

Rdatatable / data.table

Compiling on macOS 10.15.2 with OpenMP #4158