search

microsoft / LightGBM

A fast, distributed, high performance gradient boosting (GBT, GBDT, GBRT, GBM or MART) framework based on decision tree algorithms, used for ranking, classification and many other machine learning tasks.
https://lightgbm.readthedocs.io/en/latest/
MIT License
16.65k stars 3.83k forks source link

Unable to install gpu version properly #5804

Closed Oyasyhi closed 1 year ago

Oyasyhi commented 1 year ago

Description

I am trying to install the gpu support for lightgbm, however after installation I still am unable to use it in python. Would you be so kind as to assist in understanding what is going wrong.

I receive this error in python: LightGBMError: GPU Tree Learner was not enabled in this build. Please recompile with CMake option -DUSE_GPU=1

This seems to be because when I run the code:

make -j4

I receive this error:

make: *** No targets specified and no makefile found. Stop.

My build file has no makefile, and I am unsure why.

image

Reproducible example

These are my command logs: C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages>cd C:\PyEnvs\FYP\FYPEnvironment\Scripts

C:\PyEnvs\FYP\FYPEnvironment\Scripts>activate.bat

(FYPEnvironment) C:\PyEnvs\FYP\FYPEnvironment\Scripts>cd C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages

(FYPEnvironment) C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages>git clone --recursive https://github.com/microsoft/LightGBM Cloning into 'LightGBM'... remote: Enumerating objects: 29239, done. remote: Counting objects: 100% (102/102), done. remote: Compressing objects: 100% (77/77), done. remote: Total 29239 (delta 50), reused 43 (delta 21), pack-reused 29137 Receiving objects: 100% (29239/29239), 20.49 MiB | 10.61 MiB/s, done. Resolving deltas: 100% (21699/21699), done. Submodule 'include/boost/compute' (https://github.com/boostorg/compute) registered for path 'external_libs/compute' Submodule 'eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'external_libs/eigen' Submodule 'external_libs/fast_double_parser' (https://github.com/lemire/fast_double_parser.git) registered for path 'external_libs/fast_double_parser' Submodule 'external_libs/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'external_libs/fmt' Cloning into 'C:/PyEnvs/FYP/FYPEnvironment/Lib/site-packages/LightGBM/external_libs/compute'... remote: Enumerating objects: 21733, done. remote: Counting objects: 100% (5/5), done. remote: Compressing objects: 100% (4/4), done. remote: Total 21733 (delta 1), reused 3 (delta 1), pack-reused 21728 Receiving objects: 100% (21733/21733), 8.51 MiB | 9.68 MiB/s, done. Resolving deltas: 100% (17567/17567), done. Cloning into 'C:/PyEnvs/FYP/FYPEnvironment/Lib/site-packages/LightGBM/external_libs/eigen'... remote: Enumerating objects: 118861, done. remote: Counting objects: 100% (540/540), done. remote: Compressing objects: 100% (179/179), done. remote: Total 118861 (delta 362), reused 534 (delta 361), pack-reused 118321 Receiving objects: 100% (118861/118861), 103.16 MiB | 15.85 MiB/s, done. Resolving deltas: 100% (98197/98197), done. Cloning into 'C:/PyEnvs/FYP/FYPEnvironment/Lib/site-packages/LightGBM/external_libs/fast_double_parser'... remote: Enumerating objects: 781, done. remote: Counting objects: 100% (180/180), done. remote: Compressing objects: 100% (66/66), done. remote: Total 781 (delta 124), reused 131 (delta 103), pack-reused 601 Receiving objects: 100% (781/781), 833.45 KiB | 7.38 MiB/s, done. Resolving deltas: 100% (395/395), done. Cloning into 'C:/PyEnvs/FYP/FYPEnvironment/Lib/site-packages/LightGBM/external_libs/fmt'... remote: Enumerating objects: 31695, done. remote: Counting objects: 100% (51/51), done. remote: Compressing objects: 100% (24/24), done. remote: Total 31695 (delta 24), reused 38 (delta 19), pack-reused 31644 Receiving objects: 100% (31695/31695), 13.84 MiB | 10.38 MiB/s, done. Resolving deltas: 100% (21467/21467), done. Submodule path 'external_libs/compute': checked out '36350b7de849300bd3d72a05d8bf890ca405a014' Submodule path 'external_libs/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' Submodule path 'external_libs/fast_double_parser': checked out 'ace60646c02dc54c57f19d644e49a61e7e7758ec' Submodule 'benchmark/dependencies/abseil-cpp' (https://github.com/abseil/abseil-cpp.git) registered for path 'external_libs/fast_double_parser/benchmarks/dependencies/abseil-cpp' Submodule 'benchmark/dependencies/double-conversion' (https://github.com/google/double-conversion.git) registered for path 'external_libs/fast_double_parser/benchmarks/dependencies/double-conversion' Cloning into 'C:/PyEnvs/FYP/FYPEnvironment/Lib/site-packages/LightGBM/external_libs/fast_double_parser/benchmarks/dependencies/abseil-cpp'... remote: Enumerating objects: 21051, done. remote: Counting objects: 100% (129/129), done. remote: Compressing objects: 100% (88/88), done. remote: Total 21051 (delta 49), reused 112 (delta 40), pack-reused 20922 Receiving objects: 100% (21051/21051), 12.82 MiB | 10.38 MiB/s, done. Resolving deltas: 100% (16408/16408), done. Cloning into 'C:/PyEnvs/FYP/FYPEnvironment/Lib/site-packages/LightGBM/external_libs/fast_double_parser/benchmarks/dependencies/double-conversion'... remote: Enumerating objects: 1358, done. remote: Counting objects: 100% (202/202), done. remote: Compressing objects: 100% (110/110), done. remote: Total 1358 (delta 110), reused 157 (delta 84), pack-reused 1156 Receiving objects: 100% (1358/1358), 7.15 MiB | 9.35 MiB/s, done. Resolving deltas: 100% (882/882), done. Submodule path 'external_libs/fast_double_parser/benchmarks/dependencies/abseil-cpp': checked out 'd936052d32a5b7ca08b0199a6724724aea432309' Submodule path 'external_libs/fast_double_parser/benchmarks/dependencies/double-conversion': checked out 'f4cb2384efa55dee0e6652f8674b05763441ab09' Submodule path 'external_libs/fmt': checked out 'b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9'

(FYPEnvironment) C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages>cd lightgbm

(FYPEnvironment) C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\LightGBM>mkdir build

(FYPEnvironment) C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\LightGBM>cd build

(FYPEnvironment) C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\LightGBM\build>cmake -A x64 -DUSE_GPU=1 -DBOOST_ROOT=C:/local/boost_1_81_0 -DBOOST_LIBRARYDIR=C:/local/boost_1_81_0/lib64-msvc-14.3 -DOpenCL_LIBRARY="C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.1/lib/x64/OpenCL.lib" -DOpenCL_INCLUDE_DIR="C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.1/include" .. 'cmake' is not recognized as an internal or external command, operable program or batch file.

(FYPEnvironment) C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\LightGBM\build>set PATH=C:\Program Files\CMake\bin;%PATH%

(FYPEnvironment) C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\LightGBM\build>cmake Usage

cmake [options] cmake [options] cmake [options] -S -B

Specify a source directory to (re-)generate a build system for it in the current working directory. Specify an existing build directory to re-generate its build system.

Run 'cmake --help' for more information.

(FYPEnvironment) C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\LightGBM\build>cmake -A x64 -DUSE_GPU=1 -DBOOST_ROOT=C:/local/boost_1_81_0 -DBOOST_LIBRARYDIR=C:/local/boost_1_81_0/lib64-msvc-14.3 -DOpenCL_LIBRARY="C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.1/lib/x64/OpenCL.lib" -DOpenCL_INCLUDE_DIR="C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.1/include" .. -- Building for: Visual Studio 17 2022 -- Selecting Windows SDK version 10.0.19041.0 to target Windows 10.0.22621. -- The C compiler identification is MSVC 19.33.31630.0 -- The CXX compiler identification is MSVC 19.33.31630.0 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.33.31629/bin/Hostx64/x64/cl.exe - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.33.31629/bin/Hostx64/x64/cl.exe - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- Found OpenMP_C: -openmp (found version "2.0") -- Found OpenMP_CXX: -openmp (found version "2.0") -- Found OpenMP: TRUE (found version "2.0") -- Looking for CL_VERSION_3_0 -- Looking for CL_VERSION_3_0 - found -- Found OpenCL: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.1/lib/x64/OpenCL.lib (found version "3.0") -- OpenCL include directory: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.1/include -- Found Boost: C:/local/boost_1_81_0 (found suitable version "1.81.0", minimum required is "1.56.0") found components: filesystem system -- Performing Test MM_PREFETCH -- Performing Test MM_PREFETCH - Failed -- Performing Test MM_MALLOC -- Performing Test MM_MALLOC - Failed -- Looking for inet_pton -- Looking for inet_pton - found -- Configuring done (8.9s) -- Generating done (0.1s) -- Build files have been written to: C:/PyEnvs/FYP/FYPEnvironment/Lib/site-packages/LightGBM/build

(FYPEnvironment) C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\LightGBM\build>make make: *** No targets specified and no makefile found. Stop.

(FYPEnvironment) C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\LightGBM\build>make -j4 make: *** No targets specified and no makefile found. Stop.

(FYPEnvironment) C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\LightGBM\build>cmake --build . --target ALL_BUILD --config Release MSBuild version 17.3.1+2badb37d1 for .NET Framework Checking Build System Building Custom Rule C:/PyEnvs/FYP/FYPEnvironment/Lib/site-packages/LightGBM/CMakeLists.txt c_api.cpp lightgbm_capi_objs.vcxproj -> C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\LightGBM\build\lightgbm_capi_objs.dir\Re lease\lightgbm_capi_objs.lib Building Custom Rule C:/PyEnvs/FYP/FYPEnvironment/Lib/site-packages/LightGBM/CMakeLists.txt boosting.cpp cuda_score_updater.cpp gbdt.cpp gbdt_model_text.cpp gbdt_prediction.cpp prediction_early_stop.cpp sample_strategy.cpp cuda_utils.cpp bin.cpp config.cpp config_auto.cpp cuda_column_data.cpp cuda_metadata.cpp cuda_row_data.cpp cuda_tree.cpp dataset.cpp dataset_loader.cpp file_io.cpp json11.cpp metadata.cpp parser.cpp train_share_states.cpp tree.cpp cuda_binary_metric.cpp cuda_pointwise_metric.cpp cuda_regression_metric.cpp dcg_calculator.cpp metric.cpp linker_topo.cpp linkers_mpi.cpp linkers_socket.cpp network.cpp cuda_binary_objective.cpp cuda_multiclass_objective.cpp cuda_rank_objective.cpp cuda_regression_objective.cpp objective_function.cpp cuda_best_split_finder.cpp cuda_data_partition.cpp cuda_histogram_constructor.cpp cuda_leaf_splits.cpp cuda_single_gpu_tree_learner.cpp data_parallel_tree_learner.cpp feature_parallel_tree_learner.cpp gpu_tree_learner.cpp linear_tree_learner.cpp serial_tree_learner.cpp tree_learner.cpp voting_parallel_tree_learner.cpp cl_version.h: CL_TARGET_OPENCL_VERSION is not defined. Defaulting to 300 (OpenCL 3.0) cl_version.h: CL_TARGET_OPENCL_VERSION is not defined. Defaulting to 300 (OpenCL 3.0) cl_version.h: CL_TARGET_OPENCL_VERSION is not defined. Defaulting to 300 (OpenCL 3.0) cl_version.h: CL_TARGET_OPENCL_VERSION is not defined. Defaulting to 300 (OpenCL 3.0) cl_version.h: CL_TARGET_OPENCL_VERSION is not defined. Defaulting to 300 (OpenCL 3.0) lightgbm_objs.vcxproj -> C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\LightGBM\build\lightgbm_objs.dir\Release\ligh tgbm_objs.lib Building Custom Rule C:/PyEnvs/FYP/FYPEnvironment/Lib/site-packages/LightGBM/CMakeLists.txt Creating library C:/PyEnvs/FYP/FYPEnvironment/Lib/site-packages/LightGBM/Release/lib_lightgbm.lib and object C:/Py Envs/FYP/FYPEnvironment/Lib/site-packages/LightGBM/Release/lib_lightgbm.exp _lightgbm.vcxproj -> C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\LightGBM\Release\lib_lightgbm.dll Building Custom Rule C:/PyEnvs/FYP/FYPEnvironment/Lib/site-packages/LightGBM/CMakeLists.txt main.cpp application.cpp Creating library C:/PyEnvs/FYP/FYPEnvironment/Lib/site-packages/LightGBM/Release/lightgbm.lib and object C:/PyEnvs /FYP/FYPEnvironment/Lib/site-packages/LightGBM/Release/lightgbm.exp lightgbm.vcxproj -> C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\LightGBM\Release\lightgbm.exe Building Custom Rule C:/PyEnvs/FYP/FYPEnvironment/Lib/site-packages/LightGBM/CMakeLists.txt

(FYPEnvironment) C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\LightGBM\build>cd C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\LightGBM\python-package

(FYPEnvironment) C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\LightGBM\python-package>setup.py install

(FYPEnvironment) C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\LightGBM\python-package>python setup.py install running install Starting to compile the library. Starting to compile with MSBuild from existing solution file. Compilation with MSBuild from existing solution file failed. Starting to compile with Visual Studio 17 2022 (x64). running build running build_py creating build creating build\lib creating build\lib\lightgbm copying lightgbm\basic.py -> build\lib\lightgbm copying lightgbm\callback.py -> build\lib\lightgbm copying lightgbm\compat.py -> build\lib\lightgbm copying lightgbm\dask.py -> build\lib\lightgbm copying lightgbm\engine.py -> build\lib\lightgbm copying lightgbm\libpath.py -> build\lib\lightgbm copying lightgbm\plotting.py -> build\lib\lightgbm copying lightgbm\sklearn.py -> build\lib\lightgbm copying lightgbm__init.py -> build\lib\lightgbm running egg_info creating lightgbm.egg-info writing lightgbm.egg-info\PKG-INFO writing dependency_links to lightgbm.egg-info\dependency_links.txt writing requirements to lightgbm.egg-info\requires.txt writing top-level names to lightgbm.egg-info\top_level.txt writing manifest file 'lightgbm.egg-info\SOURCES.txt' reading manifest file 'lightgbm.egg-info\SOURCES.txt' reading manifest template 'MANIFEST.in' no previously-included directories found matching 'build' warning: no files found matching '.txt' warning: no files found matching '.so' under directory 'lightgbm' warning: no files found matching 'compile\cmake\IntegratedOpenCL.cmake' warning: no files found matching '.so' under directory 'compile' warning: no files found matching 'compile\external_libs\compute\CMakeLists.txt' warning: no files found matching '' under directory 'compile\external_libs\compute\cmake' warning: no files found matching '' under directory 'compile\external_libs\compute\include' warning: no files found matching '' under directory 'compile\external_libs\compute\meta' warning: no files found matching '.dll' under directory 'compile\windows\x64\DLL' warning: no previously-included files matching '.py[co]' found anywhere in distribution warning: no previously-included files found matching 'compile\external_libs\compute.git' writing manifest file 'lightgbm.egg-info\SOURCES.txt' copying lightgbm\VERSION.txt -> build\lib\lightgbm copying lightgbm\py.typed -> build\lib\lightgbm running install_lib copying build\lib\lightgbm\basic.py -> C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\lightgbm copying build\lib\lightgbm\callback.py -> C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\lightgbm copying build\lib\lightgbm\compat.py -> C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\lightgbm copying build\lib\lightgbm\dask.py -> C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\lightgbm copying build\lib\lightgbm\engine.py -> C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\lightgbm copying build\lib\lightgbm\libpath.py -> C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\lightgbm copying build\lib\lightgbm\plotting.py -> C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\lightgbm copying build\lib\lightgbm\py.typed -> C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\lightgbm copying build\lib\lightgbm\sklearn.py -> C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\lightgbm copying build\lib\lightgbm\VERSION.txt -> C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\lightgbm copying build\lib\lightgbm\init.py -> C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\lightgbm Installing lib_lightgbm from: ['C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\LightGBM\python-package\compile\Release\lib_lightgbm.dll', 'C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\LightGBM\Release\lib_lightgbm.dll'] copying C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\LightGBM\python-package\compile\Release\lib_lightgbm.dll -> C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\lightgbm byte-compiling C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\lightgbm\basic.py to basic.cpython-39.pyc byte-compiling C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\lightgbm\callback.py to callback.cpython-39.pyc byte-compiling C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\lightgbm\compat.py to compat.cpython-39.pyc byte-compiling C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\lightgbm\dask.py to dask.cpython-39.pyc byte-compiling C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\lightgbm\engine.py to engine.cpython-39.pyc byte-compiling C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\lightgbm\libpath.py to libpath.cpython-39.pyc byte-compiling C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\lightgbm\plotting.py to plotting.cpython-39.pyc byte-compiling C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\lightgbm\sklearn.py to sklearn.cpython-39.pyc byte-compiling C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\lightgbm\init.py to init__.cpython-39.pyc running install_egg_info Copying lightgbm.egg-info to C:\PyEnvs\FYP\FYPEnvironment\Lib\site-packages\lightgbm-3.3.5.99-py3.9.egg-info running install_scripts

Environment info

Python 3.9.13 CUDA Version 12.1 Nvidia GeForce RTX 3070 Driver Version: 531.41 Windows 11

Other things tried: -DUSE_CUDA=1 doesnt work either make -j4 doesnt seem to work as seen above

jameslamb commented 1 year ago

Sorry for the delayed response.

It's difficult to tell from what you've provided, but I think you missed a step (running cmake .. from inside the build/ directory). Please consult https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax to learn how to format text on GitHub.

Please also see https://github.com/microsoft/LightGBM/issues/5897#issuecomment-1567656771 for some more information on how to install the GPU-enabled version of the Python package.

github-actions[bot] commented 1 year ago

This issue has been automatically closed because it has been awaiting a response for too long. When you have time to to work with the maintainers to resolve this issue, please post a new comment and it will be re-opened. If the issue has been locked for editing by the time you return to it, please open a new issue and reference this one. Thank you for taking the time to improve LightGBM!

github-actions[bot] commented 1 year ago

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.