search

microsoft / LightGBM

A fast, distributed, high performance gradient boosting (GBT, GBDT, GBRT, GBM or MART) framework based on decision tree algorithms, used for ranking, classification and many other machine learning tasks.
https://lightgbm.readthedocs.io/en/latest/
MIT License
16.61k stars 3.83k forks source link

Failed install to Mac OS from RStudio and from terminal #1852

Closed pbpearman closed 5 years ago

pbpearman commented 5 years ago

I have been trying to install to Mac, and have tried several possible solutions, but nothing seems to work. Following issue 1407, I tried installing from the RStudio console: 

> install_github("Microsoft/LightGBM", subdir = "R-package")
ld: warning: text-based stub file /System/Library/Frameworks//CoreFoundation.framework/CoreFoundation.tbd and library file /System/Library/Frameworks//CoreFoundation.framework/CoreFoundation are out of sync. Falling back to library file for linking.
installing via 'install.libs.R' to /Library/Frameworks/R.framework/Versions/3.5/Resources/library/lightgbm
Error in eval(ei, envir) : Copying CMakeLists failed
* removing ‘/Library/Frameworks/R.framework/Versions/3.5/Resources/library/lightgbm’
Error in i.p(...) : 
(converted from warning) installation of package ‘/var/folders/bz/h_ys6bjn3h9b201nks55l9bc0000gn/T//Rtmp9TFE7v/file8de29d49380/lightgbm_2.2.3.tar.gz’ had non-zero exit status

I also tried from within the LightGBM folder and got completely a different error message:

> Rscript build_r.R
Updating lightgbm documentation
Updating roxygen version in /Users/bgppermp/Downloads/LightGBM/lightgbm_r/DESCRIPTION
Writing NAMESPACE
Loading lightgbm
Loading required package: R6
Error in getDLLRegisteredRoutines.DLLInfo(dll, addNames = FALSE) : 
must specify DLL via a “DLLInfo” object. See getLoadedDLLs()
Calls: <Anonymous> ... assignNativeRoutines -> getDLLRegisteredRoutines.DLLInfo
In addition: Warning message:
roxygen2 requires Encoding: UTF-8 
Execution halted

So, I'm at a dead end in both cases. I have tried to set the CXX and CC flags to (g++ or g++-8) and (gcc and gcc-8). Your suggestions would be appreciated because with the differing error messages, this is now beyond me.

Environment info

Operating System: Mac OS X 10.13.6

CPU/GPU model: 3 GHz 8-Core Intel Xeon E5

C++/Python/R version: R v. 3.5.1

rvm_bin_path=/Users/bgppermp/.rvm/bin
TERM_PROGRAM=Apple_Terminal
GEM_HOME=/Users/bgppermp/.rvm/gems/ruby-2.4.2
SHELL=/bin/bash
TERM=xterm-256color
IRBRC=/Users/bgppermp/.rvm/rubies/ruby-2.4.2/.irbrc
TMPDIR=/var/folders/bz/h_ys6bjn3h9b201nks55l9bc0000gn/T/
Apple_PubSub_Socket_Render=/private/tmp/com.apple.launchd.nAzjTbF0wY/Render
TERM_PROGRAM_VERSION=404
OLDPWD=/Users/bgppermp/Downloads/LightGBM
MY_RUBY_HOME=/Users/bgppermp/.rvm/rubies/ruby-2.4.2
TERM_SESSION_ID=7A5FF598-D214-4F1C-A10E-D177BA13BFB4
USER=bgppermp
_system_type=Darwin
rvm_path=/Users/bgppermp/.rvm
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.wAGuhJZysJ/Listeners
rvm_prefix=/Users/bgppermp
PATH=/usr/local/opt/curl/bin:/usr/local/bin:/Users/bgppermp/.rvm/gems/ruby-2.4.2/bin:/Users/bgppermp/.rvm/gems/ruby-2.4.2@global/bin:/Users/bgppermp/.rvm/rubies/ruby-2.4.2/bin:/usr/local/opt/openssl@1.1/bin:/Users/bgppermp/anaconda/bin:/opt/local/bin:/opt/local/sbin:/Library/Frameworks/GDAL.framework/Programs:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/local/ncbi/blast/bin:/Library/TeX/texbin:/usr/local/MacGPG2/bin:/opt/X11/bin:~/bin:/Users/bgppermp/.rvm/bin
PWD=/Users/bgppermp/Downloads/LightGBM/R-package/src
LANG=en_US.UTF-8
_system_arch=x86_64
XPC_FLAGS=0x0
BLASTDB=/Volumes/lev_pearman/blastdb
_system_version=10.13
CXX=/usr/local/bin/g++
XPC_SERVICE_NAME=0
rvm_version=1.29.3 (latest)
SHLVL=1
HOME=/Users/bgppermp
LOGNAME=bgppermp
GEM_PATH=/Users/bgppermp/.rvm/gems/ruby-2.4.2:/Users/bgppermp/.rvm/gems/ruby-2.4.2@global
CC=/usr/local/bin/gcc
DISPLAY=/private/tmp/com.apple.launchd.gPxyTK0544/org.macosforge.xquartz:0
RUBY_VERSION=ruby-2.4.2
SECURITYSESSIONID=186a9
_system_name=OSX
_=/usr/bin/env

Error message

Reproducible examples

Steps to reproduce

1. 2. 3.

pbpearman commented 5 years ago

Update. Following this post and adding some links, I got a successful build following this post. I do: `bgppermp$ cd R-package/ uhtred:R-package bgppermp$ R CMD INSTALL --build . --no-multiarch

Laurae2 commented 5 years ago

@pbpearman Did you build the package first using the build_r.R script? (it builds and installs, actually)

pbpearman commented 5 years ago

@Laurae2 Yes, I tried, but it doesn't complete. In the second example in the original post, I showed what happened when using that script from within the LightGBM folder. The fairly cryptic error message was: Error in getDLLRegisteredRoutines.DLLInfo(dll, addNames = FALSE) : must specify DLL via a “DLLInfo” object. See getLoadedDLLs() This might provide another approach to trouble-shooting, but I looked at the build_r.R script and did not find anything I could relate to the error message. I tried again from a fresh clone, but got the same result.

thebamoor commented 5 years ago

Tried to install LightGBM; kept on getting "Copying CMakeLists failed" messages

ben519 commented 5 years ago

I too am getting this error, on mac osx 10.14.1 running R 3.5.1. Perhaps this is related to #1702?

Bens-MacBook-Pro-2017:LightGBM Ben$ Rscript build_r.R
Updating lightgbm documentation
Updating roxygen version in /Users/Ben/Documents/Projects/R/LightGBM/lightgbm_r/DESCRIPTION
Writing NAMESPACE
Loading lightgbm
Loading required package: R6
Error in getDLLRegisteredRoutines.DLLInfo(dll, addNames = FALSE) : 
  must specify DLL via a “DLLInfo” object. See getLoadedDLLs()
Calls: <Anonymous> ... assignNativeRoutines -> getDLLRegisteredRoutines.DLLInfo
In addition: Warning message:
roxygen2 requires Encoding: UTF-8 
Execution halted
Bens-MacBook-Pro-2017:LightGBM Ben$
Laurae2 commented 5 years ago

Ping @jameslamb : DLL on Mac or generic error?

Laurae2 commented 5 years ago

@pbpearman Can you try lgbdl to see if it provides an another error message? Make sure to use a fresh R session and to close all other R sessions.

ben519 commented 5 years ago

Hmm, it seems that the line devtools::document(pkg = "lightgbm_r") in build_r.R is the culprit. If I run the script without that line, lightgbm appears to install successfully.

Laurae2 commented 5 years ago

@ben519 Sounds like the R session is not cleaned after generating the documentation because this DLL error appears when trying to install a package with a corresponding .so/.dll already loaded in memory (here, installing LightGBM while LightGBM is being used).

pbpearman commented 5 years ago

@Laurae2, @ben519 Here is what I get when running lgbdl from a fresh R session: `> lgbdl::lgb.dl(compiler = "gcc") Cloning into 'LightGBM'... Submodule 'include/boost/compute' (https://github.com/boostorg/compute) registered for path 'compute' Cloning into '/private/var/folders/bz/h_ys6bjn3h9b201nks55l9bc0000gn/T/Rtmpw8LARd/LightGBM/compute'... Submodule path 'compute': checked out '509ebe4a9282eec8a92c65ce3bbc1925f1fdbe07' Already on 'master' Your branch is up to date with 'origin/master'.

-- Configuring incomplete, errors occurred! See also "/private/var/folders/bz/h_ys6bjn3h9b201nks55l9bc0000gn/T/Rtmpw8LARd/LightGBM/lightgbm_r/src/build/CMakeFiles/CMakeOutput.log". make: *** No rule to make target `_lightgbm'. Stop. Error in eval(ei, envir) : Cannot find lib_lightgbm.so

Also, I note this: bgppermp$ echo |cpp -fopenmp -dM |grep -i open

define _OPENMP 201511

jameslamb commented 5 years ago

@pbpearman it looks like clang is not finding openmp. Can you humor me and try this as we note in the installation guide?

brew install libomp

I also have never tried from within RStudio but given my experience with that tool, I echo @Laurae2 's concern that something is being left behind.

If you run

Rscript build_r.R

from the command line, this would not be a concern because every Rscript call is a new R session by definition.

1psmith commented 5 years ago

I'm getting the same thing both with the command line and with Rstudio. (Windows install)

C:\LightGBM>Rscript build_r.R Updating lightgbm documentation Updating roxygen version in C:\LightGBM\lightgbm_r/DESCRIPTION Writing NAMESPACE Loading lightgbm Loading required package: R6 Error in getDLLRegisteredRoutines.DLLInfo(dll, addNames = FALSE) : must specify DLL via a "DLLInfo" object. See getLoadedDLLs() Calls: ... assignNativeRoutines -> getDLLRegisteredRoutines.DLLInfo In addition: Warning message: roxygen2 requires Encoding: UTF-8 Execution halted

1psmith commented 5 years ago

Update: Ran this on my digital ocean Ubuntu 18.04 server and got exactly the same thing.

$ Rscript build_r.R Updating lightgbm documentation Updating roxygen version in /home/pres/LightGBM/lightgbm_r/DESCRIPTION Writing NAMESPACE Loading lightgbm Loading required package: R6 Error in getDLLRegisteredRoutines.DLLInfo(dll, addNames = FALSE) : must specify DLL via a “DLLInfo” object. See getLoadedDLLs() Calls: ... assignNativeRoutines -> getDLLRegisteredRoutines.DLLInfo In addition: Warning message: roxygen2 requires Encoding: UTF-8 Execution halted

Laurae2 commented 5 years ago

@1psmith Comment the line 49 of build_r.R, I am also getting the error.

Laurae2 commented 5 years ago

@1psmith You can try also #1868 fix.

git clone --recursive https://github.com/Microsoft/LightGBM
cd LightGBM
git checkout fbd98d5
Rscript build_r.R
1psmith commented 5 years ago

Update: Ran this on my digital ocean Ubuntu 18.04 server and got exactly the same thing.

$ Rscript build_r.R Updating lightgbm documentation Updating roxygen version in /home/pres/LightGBM/lightgbm_r/DESCRIPTION Writing NAMESPACE Loading lightgbm Loading required package: R6 Error in getDLLRegisteredRoutines.DLLInfo(dll, addNames = FALSE) : must specify DLL via a “DLLInfo” object. See getLoadedDLLs() Calls: ... assignNativeRoutines -> getDLLRegisteredRoutines.DLLInfo In addition: Warning message: roxygen2 requires Encoding: UTF-8 Execution halted

@1psmith You can try also #1868 fix.

git clone --recursive https://github.com/Microsoft/LightGBM
cd LightGBM
git checkout fbd98d5
Rscript build_r.R

Yes! This worked perfectly. Thanks!!!

pbpearman commented 5 years ago

@Laurae2 @jameslamb @1psmith Here, for entertainment, I installed libomp anew. Then I try @1psmith 's idea. `bgppermp$ brew install libomp Updating Homebrew... ==> Auto-updated Homebrew! Updated 2 taps (homebrew/core and homebrew/cask). ==> Updated Formulae jump laszip mkl-dnn vsts-cli vtk youtube-dl

==> Downloading https://homebrew.bintray.com/bottles/libomp-6.0.1.high_sierra.bo ==> Downloading from https://akamai.bintray.com/04/046f659ad8a2cc336049a1e7f7be0 ######################################################################## 100.0% ==> Pouring libomp-6.0.1.high_sierra.bottle.tar.gz Error: The brew link step did not complete successfully The formula built, but is not symlinked into /usr/local Could not symlink lib/libgomp.a Target /usr/local/lib/libgomp.a already exists. You may want to remove it: rm '/usr/local/lib/libgomp.a'

To force the link and overwrite all conflicting files: brew link --overwrite libomp

To list all files that would be deleted: brew link --overwrite --dry-run libomp

Possible conflicting files are: /usr/local/lib/libgomp.a /usr/local/lib/libgomp.dylib -> /usr/local/lib/libomp.dylib /usr/local/lib/libiomp5.dylib -> /usr/local/lib/libomp.dylib /usr/local/lib/libomp.dylib ==> Caveats On Apple Clang, you need to add several options to use OpenMP's front end instead of the standard driver option. This usually looks like -Xpreprocessor -fopenmp -lomp

You might need to make sure the lib and include directories are discoverable if /usr/local is not searched:

-L/usr/local/opt/libomp/lib -I/usr/local/opt/libomp/include

For CMake, the following flags will cause the OpenMP::OpenMP_CXX target to be set up correctly: -DOpenMP_CXX_FLAGS="-Xpreprocessor -fopenmp -I/usr/local/opt/libomp/include" -DOpenMP_CXX_LIB_NAMES="omp" -DOpenMP_omp_LIBRARY=/usr/local/opt/libomp/lib/libomp.dylib ==> Summary 🍺 /usr/local/Cellar/libomp/6.0.1: 12 files, 1.2MB

bgppermp$ brew link --overwrite libomp Linking /usr/local/Cellar/libomp/6.0.1... 8 symlinks created

bgppermp$ git checkout fbd98d5 Note: checking out 'fbd98d5'.

You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may do so (now or later) by using -b with the checkout command again. Example:

git checkout -b

HEAD is now at fbd98d5 Does not need to fix doc. Fixes #1852

bgppermp$ git branch

Worked! Although the link to libomp was present, re-installing and overwriting the old link seems to have fixed the situation. Must have been broken. Thanks!

`

ben519 commented 5 years ago

For reference, I think the root of this issue is discussed here.