Closed thvasilo closed 2 years ago
Thanks for the excellent reproducible example!
pytest tests/python_package_test
? There are some other tests in the tests/
directory which are not for the Python package, and expect that LightGBM was compiled differently.See https://github.com/microsoft/LightGBM/blob/8f4126d66065347519185b2f687d423117d67b1b/.ci/test.sh#L132-L139 for an example of how LightGBM's Python package tests are run.
Sure, here's a full output of pytest tests/python_package_test/
: https://gist.github.com/thvasilo/52702cdcff9d8c9d4659f1b2885f0b1c
Thanks very much @thvasilo ! Sorry for the delayed response.
I see the following in the logs you provided.
__________ ERROR collecting tests/python_package_test/test_engine.py ___________
ImportError while importing test module '/Users/XXX/repos/LightGBM/tests/python_package_test/test_engine.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../opt/anaconda3/envs/lgbm-dev/lib/python3.9/importlib/__init__.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/python_package_test/test_engine.py:12: in <module>
import psutil
E ModuleNotFoundError: No module named 'psutil'
psutil
is imported unconditionally in one of the tests files, so it needs to be installed to run the tests.
I don't necessarily think that's related to the other OSError
, but it's worth addressing, in case it is related in some non-obvious way.
I personally don't have an M1 Mac, and LightGBM doesn't have any CI jobs testing on one yet, so unless another maintainer has an M1 Mac or you are willing to help us investigate further, I'm not sure how to make progress on this issue at the moment.
We do have a feature request for this (#3606) but I'm not aware of anyone actively working on it, because of those practical limitations.
Linking this related issue tracking support for M1 Mac environment on GitHub Actions: https://github.com/actions/virtual-environments/issues/2187
Other things I might try:
cmake
cmake
3.22.x. The M1 Mac stuff has been moving fast, so it's possible some significant developments have happened in newer releasesI strongly believe that @thvasilo is right and this is
a common problem with other Python packages and M1 Macs
and we can't do anything with it. Probably this affects Python 3.9 version only.
Some random examples:
@thvasilo Also, raw logs may help. Please install LightGBM in the following way:
brew install cmake
brew install libomp
git clone --recursive https://github.com/microsoft/LightGBM
cd LightGBM
mkdir build
cd build
cmake ..
make -j4
according to https://lightgbm.readthedocs.io/en/latest/Installation-Guide.html#build-from-github and after that run
cd ../python-package
python setup.py install --precompile
I tried using the pre-compiled libraries, but not sure it's working as intended:
running install
running build
running build_py
running egg_info
writing lightgbm.egg-info/PKG-INFO
writing dependency_links to lightgbm.egg-info/dependency_links.txt
writing requirements to lightgbm.egg-info/requires.txt
writing top-level names to lightgbm.egg-info/top_level.txt
reading manifest file 'lightgbm.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
no previously-included directories found matching 'build'
warning: no files found matching '*.txt'
warning: no files found matching '*.so' under directory 'lightgbm'
warning: no files found matching 'compile/cmake/IntegratedOpenCL.cmake'
warning: no files found matching '*.dll' under directory 'compile/Release'
warning: no files found matching 'compile/external_libs/compute/CMakeLists.txt'
warning: no files found matching '*' under directory 'compile/external_libs/compute/cmake'
warning: no files found matching '*' under directory 'compile/external_libs/compute/include'
warning: no files found matching '*' under directory 'compile/external_libs/compute/meta'
warning: no files found matching '*.dll' under directory 'compile/windows/x64/DLL'
warning: no previously-included files matching '*.py[co]' found anywhere in distribution
warning: no previously-included files found matching 'compile/external_libs/compute/.git'
adding license file 'LICENSE'
writing manifest file 'lightgbm.egg-info/SOURCES.txt'
copying lightgbm/VERSION.txt -> build/lib/lightgbm
running install_lib
copying build/lib/lightgbm/VERSION.txt -> /Users/thvasilo/opt/anaconda3/envs/lgb-unb/lib/python3.9/site-packages/lightgbm
INFO:LightGBM:Installing lib_lightgbm from: ['/Users/thvasilo/repos/LightGBM/lib_lightgbm.so', '/Users/thvasilo/repos/LightGBM/python-package/compile/lib_lightgbm.so']
running install_egg_info
removing '/Users/thvasilo/opt/anaconda3/envs/lgb-unb/lib/python3.9/site-packages/lightgbm-3.3.1.99-py3.9.egg-info' (and everything under it)
Copying lightgbm.egg-info to /Users/thvasilo/opt/anaconda3/envs/lgb-unb/lib/python3.9/site-packages/lightgbm-3.3.1.99-py3.9.egg-info
running install_scripts
I've also tried including the CMake -DCMAKE_OSX_ARCHITECTURES=arm64
option within setup.py with a small hack like so:
Index: python-package/setup.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/python-package/setup.py b/python-package/setup.py
--- a/python-package/setup.py (revision 0cbe02d27fa94495a123e80131871e772dd38d45)
+++ b/python-package/setup.py (date 1641937361373)
@@ -26,6 +26,7 @@
('hdfs', 'h', 'Compile HDFS version'),
('bit32', None, 'Compile 32-bit version'),
('precompile', 'p', 'Use precompiled library'),
+ ('arm', None, 'Build for arm64 arch'),
('boost-root=', None, 'Boost preferred installation prefix'),
('boost-dir=', None, 'Directory with Boost package configuration file'),
('boost-include-dir=', None, 'Directory containing Boost headers'),
@@ -114,7 +115,8 @@
opencl_library: Optional[str] = None,
nomp: bool = False,
bit32: bool = False,
- integrated_opencl: bool = False
+ integrated_opencl: bool = False,
+ arm: bool = False
) -> None:
build_dir = CURRENT_DIR / "build_cpp"
rmtree(build_dir, ignore_errors=True)
@@ -150,6 +152,8 @@
cmake_cmd.append("-DUSE_OPENMP=OFF")
if use_hdfs:
cmake_cmd.append("-DUSE_HDFS=ON")
+ if arm:
+ cmake_cmd.append("-DCMAKE_OSX_ARCHITECTURES=arm64")
if system() in {'Windows', 'Microsoft'}:
if use_mingw:
@@ -238,6 +242,7 @@
self.precompile = False
self.nomp = False
self.bit32 = False
+ self.arm = False
def run(self) -> None:
if (8 * struct.calcsize("P")) != 64:
@@ -254,7 +259,7 @@
use_hdfs=self.hdfs, boost_root=self.boost_root, boost_dir=self.boost_dir,
boost_include_dir=self.boost_include_dir, boost_librarydir=self.boost_librarydir,
opencl_include_dir=self.opencl_include_dir, opencl_library=self.opencl_library,
- nomp=self.nomp, bit32=self.bit32, integrated_opencl=self.integrated_opencl)
+ nomp=self.nomp, bit32=self.bit32, integrated_opencl=self.integrated_opencl, arm=self.arm)
install.run(self)
if LOG_PATH.is_file():
LOG_PATH.unlink()
Still getting the same error with the above though.
@thvasilo
I tried using the pre-compiled libraries, but not sure it's working as intended:
Is it possible to share installation logs from CMake (before python setup.py install --precompile
)?
Sure, could you tell me which ones you mean exactly?
I ran
mkdir build
cd build
cmake ..
make all -j8
cd ../python-package
python setup.py install --precompile
The output of cmake ..
was:
-- The C compiler identification is AppleClang 13.0.0.13000029
-- The CXX compiler identification is AppleClang 13.0.0.13000029
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Using _mm_malloc
-- Could NOT find GTest (missing: GTest_DIR)
-- Did not find Google Test in the system root. Fetching Google Test now...
-- Found Python: /Users/thvasilo/opt/anaconda3/envs/lgb-unb/bin/python3.9 (found version "3.9.7") found components: Interpreter
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/thvasilo/repos/LightGBM/build
Thanks, I was asking for logs from cmake ..
.
The output you've provided is quite weird. There is no _mm_prefetch
check, Google Test is built for some reason, for example...
Here is an example of normal cmake ...
at macOS:
-- The C compiler identification is AppleClang 10.0.1.10010046
-- The CXX compiler identification is AppleClang 10.0.1.10010046
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Applications/Xcode_10.3.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Applications/Xcode_10.3.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found OpenMP_C: -Xclang -fopenmp (found version "3.1")
-- Found OpenMP_CXX: -Xclang -fopenmp (found version "3.1")
-- Found OpenMP: TRUE (found version "3.1")
-- Performing Test MM_PREFETCH
-- Performing Test MM_PREFETCH - Success
-- Using _mm_prefetch
-- Performing Test MM_MALLOC
-- Performing Test MM_MALLOC - Success
-- Using _mm_malloc
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/runner/work/1/s/build
I probably ran cmake previously on that instance. Here it is on a clean build
dir:
cmake ..
-- The C compiler identification is AppleClang 13.0.0.13000029
-- The CXX compiler identification is AppleClang 13.0.0.13000029
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found OpenMP_C: -Xclang -fopenmp (found version "5.0")
-- Found OpenMP_CXX: -Xclang -fopenmp (found version "5.0")
-- Found OpenMP: TRUE (found version "5.0")
-- Performing Test MM_PREFETCH
-- Performing Test MM_PREFETCH - Success
-- Using _mm_prefetch
-- Performing Test MM_MALLOC
-- Performing Test MM_MALLOC - Success
-- Using _mm_malloc
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/thvasilo/repos/LightGBM/build
OK, these logs are much better.
If you ran
make all -j8
cd ../python-package
python setup.py install --precompile
after that, you still get that original no suitable image found
error?
I did the above on a fresh clone of the library, same result. Here's the output of python setup.py install --precompile
running install
running build
running build_py
creating build
creating build/lib
creating build/lib/lightgbm
copying lightgbm/callback.py -> build/lib/lightgbm
copying lightgbm/compat.py -> build/lib/lightgbm
copying lightgbm/plotting.py -> build/lib/lightgbm
copying lightgbm/__init__.py -> build/lib/lightgbm
copying lightgbm/engine.py -> build/lib/lightgbm
copying lightgbm/dask.py -> build/lib/lightgbm
copying lightgbm/basic.py -> build/lib/lightgbm
copying lightgbm/libpath.py -> build/lib/lightgbm
copying lightgbm/sklearn.py -> build/lib/lightgbm
running egg_info
creating lightgbm.egg-info
writing lightgbm.egg-info/PKG-INFO
writing dependency_links to lightgbm.egg-info/dependency_links.txt
writing requirements to lightgbm.egg-info/requires.txt
writing top-level names to lightgbm.egg-info/top_level.txt
writing manifest file 'lightgbm.egg-info/SOURCES.txt'
reading manifest file 'lightgbm.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
no previously-included directories found matching 'build'
warning: no files found matching 'LICENSE'
warning: no files found matching '*.txt'
warning: no files found matching '*.so' under directory 'lightgbm'
warning: no files found matching 'compile/CMakeLists.txt'
warning: no files found matching 'compile/cmake/IntegratedOpenCL.cmake'
warning: no files found matching '*.so' under directory 'compile'
warning: no files found matching '*.dll' under directory 'compile/Release'
warning: no files found matching 'compile/external_libs/compute/CMakeLists.txt'
warning: no files found matching '*' under directory 'compile/external_libs/compute/cmake'
warning: no files found matching '*' under directory 'compile/external_libs/compute/include'
warning: no files found matching '*' under directory 'compile/external_libs/compute/meta'
warning: no files found matching 'compile/external_libs/eigen/CMakeLists.txt'
warning: no files found matching 'compile/external_libs/eigen/Eigen/Cholesky'
warning: no files found matching 'compile/external_libs/eigen/Eigen/Core'
warning: no files found matching 'compile/external_libs/eigen/Eigen/Dense'
warning: no files found matching 'compile/external_libs/eigen/Eigen/Eigenvalues'
warning: no files found matching 'compile/external_libs/eigen/Eigen/Geometry'
warning: no files found matching 'compile/external_libs/eigen/Eigen/Householder'
warning: no files found matching 'compile/external_libs/eigen/Eigen/Jacobi'
warning: no files found matching 'compile/external_libs/eigen/Eigen/LU'
warning: no files found matching 'compile/external_libs/eigen/Eigen/QR'
warning: no files found matching 'compile/external_libs/eigen/Eigen/SVD'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/Cholesky'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/Core'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/Eigenvalues'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/Geometry'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/Householder'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/Jacobi'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/LU'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/misc'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/plugins'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/QR'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/SVD'
warning: no files found matching 'compile/external_libs/fast_double_parser/CMakeLists.txt'
warning: no files found matching 'compile/external_libs/fast_double_parser/LICENSE'
warning: no files found matching 'compile/external_libs/fast_double_parser/LICENSE.BSL'
warning: no files found matching '*' under directory 'compile/external_libs/fast_double_parser/include'
warning: no files found matching 'compile/external_libs/fmt/CMakeLists.txt'
warning: no files found matching 'compile/external_libs/fmt/LICENSE.rst'
warning: no files found matching '*' under directory 'compile/external_libs/fmt/include'
warning: no files found matching '*' under directory 'compile/include'
warning: no files found matching '*' under directory 'compile/src'
warning: no files found matching 'LightGBM.sln' under directory 'compile/windows'
warning: no files found matching 'LightGBM.vcxproj' under directory 'compile/windows'
warning: no files found matching '*.dll' under directory 'compile/windows/x64/DLL'
warning: no previously-included files matching '*.py[co]' found anywhere in distribution
warning: no previously-included files found matching 'compile/external_libs/compute/.git'
writing manifest file 'lightgbm.egg-info/SOURCES.txt'
copying lightgbm/VERSION.txt -> build/lib/lightgbm
running install_lib
copying build/lib/lightgbm/callback.py -> /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm
copying build/lib/lightgbm/compat.py -> /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm
copying build/lib/lightgbm/plotting.py -> /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm
copying build/lib/lightgbm/__init__.py -> /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm
copying build/lib/lightgbm/engine.py -> /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm
copying build/lib/lightgbm/dask.py -> /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm
copying build/lib/lightgbm/basic.py -> /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm
copying build/lib/lightgbm/libpath.py -> /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm
copying build/lib/lightgbm/sklearn.py -> /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm
copying build/lib/lightgbm/VERSION.txt -> /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm
INFO:LightGBM:Installing lib_lightgbm from: ['/Users/thvasilo/repos/lgbm-master/lib_lightgbm.so']
copying /Users/thvasilo/repos/lgbm-master/lib_lightgbm.so -> /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm
byte-compiling /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/callback.py to callback.cpython-38.pyc
byte-compiling /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/compat.py to compat.cpython-38.pyc
byte-compiling /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/plotting.py to plotting.cpython-38.pyc
byte-compiling /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/__init__.py to __init__.cpython-38.pyc
byte-compiling /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/engine.py to engine.cpython-38.pyc
byte-compiling /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/dask.py to dask.cpython-38.pyc
byte-compiling /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/basic.py to basic.cpython-38.pyc
byte-compiling /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/libpath.py to libpath.cpython-38.pyc
byte-compiling /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/sklearn.py to sklearn.cpython-38.pyc
running install_egg_info
Copying lightgbm.egg-info to /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm-3.3.2.99-py3.8.egg-info
running install_scripts
And here is pytest:
tests $ pytest
======================================================================================== test session starts ========================================================================================
platform darwin -- Python 3.8.8, pytest-6.2.3, py-1.10.0, pluggy-0.13.1
rootdir: /Users/thvasilo/repos/lgbm-master/tests
plugins: anyio-2.2.0
collected 0 items / 9 errors
============================================================================================== ERRORS ===============================================================================================
_______________________________________________________________________________ ERROR collecting c_api_test/test_.py ________________________________________________________________________________
c_api_test/test_.py:45: in <module>
LIB = LoadDll()
c_api_test/test_.py:41: in LoadDll
lib = ctypes.cdll.LoadLibrary(lib_path[0])
../../../opt/anaconda3/lib/python3.8/ctypes/__init__.py:459: in LoadLibrary
return self._dlltype(name)
../../../opt/anaconda3/lib/python3.8/ctypes/__init__.py:381: in __init__
self._handle = _dlopen(self._name, mode)
E OSError: dlopen(/Users/thvasilo/repos/lgbm-master/lib_lightgbm.so, 6): no suitable image found. Did find:
E /Users/thvasilo/repos/lgbm-master/lib_lightgbm.so: mach-o, but wrong architecture
E /Users/thvasilo/repos/lgbm-master/lib_lightgbm.so: mach-o, but wrong architecture
________________________________________________________________________ ERROR collecting python_package_test/test_basic.py _________________________________________________________________________
python_package_test/test_basic.py:13: in <module>
import lightgbm as lgb
../../../opt/anaconda3/lib/python3.8/site-packages/lightgbm/__init__.py:8: in <module>
from .basic import Booster, Dataset, Sequence, register_logger
../../../opt/anaconda3/lib/python3.8/site-packages/lightgbm/basic.py:111: in <module>
_LIB = _load_lib()
../../../opt/anaconda3/lib/python3.8/site-packages/lightgbm/basic.py:102: in _load_lib
lib = ctypes.cdll.LoadLibrary(lib_path[0])
../../../opt/anaconda3/lib/python3.8/ctypes/__init__.py:459: in LoadLibrary
return self._dlltype(name)
../../../opt/anaconda3/lib/python3.8/ctypes/__init__.py:381: in __init__
self._handle = _dlopen(self._name, mode)
E OSError: dlopen(/Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/lib_lightgbm.so, 6): no suitable image found. Did find:
E /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/lib_lightgbm.so: mach-o, but wrong architecture
E /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/lib_lightgbm.so: mach-o, but wrong architecture
_____________________________________________________________________ ERROR collecting python_package_test/test_consistency.py ______________________________________________________________________
python_package_test/test_consistency.py:7: in <module>
import lightgbm as lgb
../../../opt/anaconda3/lib/python3.8/site-packages/lightgbm/__init__.py:8: in <module>
from .basic import Booster, Dataset, Sequence, register_logger
../../../opt/anaconda3/lib/python3.8/site-packages/lightgbm/basic.py:111: in <module>
_LIB = _load_lib()
../../../opt/anaconda3/lib/python3.8/site-packages/lightgbm/basic.py:102: in _load_lib
lib = ctypes.cdll.LoadLibrary(lib_path[0])
../../../opt/anaconda3/lib/python3.8/ctypes/__init__.py:459: in LoadLibrary
return self._dlltype(name)
../../../opt/anaconda3/lib/python3.8/ctypes/__init__.py:381: in __init__
self._handle = _dlopen(self._name, mode)
E OSError: dlopen(/Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/lib_lightgbm.so, 6): no suitable image found. Did find:
E /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/lib_lightgbm.so: mach-o, but wrong architecture
E /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/lib_lightgbm.so: mach-o, but wrong architecture
_________________________________________________________________________ ERROR collecting python_package_test/test_dask.py _________________________________________________________________________
python_package_test/test_dask.py:16: in <module>
import lightgbm as lgb
../../../opt/anaconda3/lib/python3.8/site-packages/lightgbm/__init__.py:8: in <module>
from .basic import Booster, Dataset, Sequence, register_logger
../../../opt/anaconda3/lib/python3.8/site-packages/lightgbm/basic.py:111: in <module>
_LIB = _load_lib()
../../../opt/anaconda3/lib/python3.8/site-packages/lightgbm/basic.py:102: in _load_lib
lib = ctypes.cdll.LoadLibrary(lib_path[0])
../../../opt/anaconda3/lib/python3.8/ctypes/__init__.py:459: in LoadLibrary
return self._dlltype(name)
../../../opt/anaconda3/lib/python3.8/ctypes/__init__.py:381: in __init__
self._handle = _dlopen(self._name, mode)
E OSError: dlopen(/Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/lib_lightgbm.so, 6): no suitable image found. Did find:
E /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/lib_lightgbm.so: mach-o, but wrong architecture
E /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/lib_lightgbm.so: mach-o, but wrong architecture
_________________________________________________________________________ ERROR collecting python_package_test/test_dual.py _________________________________________________________________________
python_package_test/test_dual.py:9: in <module>
import lightgbm as lgb
../../../opt/anaconda3/lib/python3.8/site-packages/lightgbm/__init__.py:8: in <module>
from .basic import Booster, Dataset, Sequence, register_logger
../../../opt/anaconda3/lib/python3.8/site-packages/lightgbm/basic.py:111: in <module>
_LIB = _load_lib()
../../../opt/anaconda3/lib/python3.8/site-packages/lightgbm/basic.py:102: in _load_lib
lib = ctypes.cdll.LoadLibrary(lib_path[0])
../../../opt/anaconda3/lib/python3.8/ctypes/__init__.py:459: in LoadLibrary
return self._dlltype(name)
../../../opt/anaconda3/lib/python3.8/ctypes/__init__.py:381: in __init__
self._handle = _dlopen(self._name, mode)
E OSError: dlopen(/Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/lib_lightgbm.so, 6): no suitable image found. Did find:
E /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/lib_lightgbm.so: mach-o, but wrong architecture
E /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/lib_lightgbm.so: mach-o, but wrong architecture
________________________________________________________________________ ERROR collecting python_package_test/test_engine.py ________________________________________________________________________
python_package_test/test_engine.py:19: in <module>
import lightgbm as lgb
../../../opt/anaconda3/lib/python3.8/site-packages/lightgbm/__init__.py:8: in <module>
from .basic import Booster, Dataset, Sequence, register_logger
../../../opt/anaconda3/lib/python3.8/site-packages/lightgbm/basic.py:111: in <module>
_LIB = _load_lib()
../../../opt/anaconda3/lib/python3.8/site-packages/lightgbm/basic.py:102: in _load_lib
lib = ctypes.cdll.LoadLibrary(lib_path[0])
../../../opt/anaconda3/lib/python3.8/ctypes/__init__.py:459: in LoadLibrary
return self._dlltype(name)
../../../opt/anaconda3/lib/python3.8/ctypes/__init__.py:381: in __init__
self._handle = _dlopen(self._name, mode)
E OSError: dlopen(/Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/lib_lightgbm.so, 6): no suitable image found. Did find:
E /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/lib_lightgbm.so: mach-o, but wrong architecture
E /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/lib_lightgbm.so: mach-o, but wrong architecture
_______________________________________________________________________ ERROR collecting python_package_test/test_plotting.py _______________________________________________________________________
python_package_test/test_plotting.py:5: in <module>
import lightgbm as lgb
../../../opt/anaconda3/lib/python3.8/site-packages/lightgbm/__init__.py:8: in <module>
from .basic import Booster, Dataset, Sequence, register_logger
../../../opt/anaconda3/lib/python3.8/site-packages/lightgbm/basic.py:111: in <module>
_LIB = _load_lib()
../../../opt/anaconda3/lib/python3.8/site-packages/lightgbm/basic.py:102: in _load_lib
lib = ctypes.cdll.LoadLibrary(lib_path[0])
../../../opt/anaconda3/lib/python3.8/ctypes/__init__.py:459: in LoadLibrary
return self._dlltype(name)
../../../opt/anaconda3/lib/python3.8/ctypes/__init__.py:381: in __init__
self._handle = _dlopen(self._name, mode)
E OSError: dlopen(/Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/lib_lightgbm.so, 6): no suitable image found. Did find:
E /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/lib_lightgbm.so: mach-o, but wrong architecture
E /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/lib_lightgbm.so: mach-o, but wrong architecture
_______________________________________________________________________ ERROR collecting python_package_test/test_sklearn.py ________________________________________________________________________
python_package_test/test_sklearn.py:19: in <module>
import lightgbm as lgb
../../../opt/anaconda3/lib/python3.8/site-packages/lightgbm/__init__.py:8: in <module>
from .basic import Booster, Dataset, Sequence, register_logger
../../../opt/anaconda3/lib/python3.8/site-packages/lightgbm/basic.py:111: in <module>
_LIB = _load_lib()
../../../opt/anaconda3/lib/python3.8/site-packages/lightgbm/basic.py:102: in _load_lib
lib = ctypes.cdll.LoadLibrary(lib_path[0])
../../../opt/anaconda3/lib/python3.8/ctypes/__init__.py:459: in LoadLibrary
return self._dlltype(name)
../../../opt/anaconda3/lib/python3.8/ctypes/__init__.py:381: in __init__
self._handle = _dlopen(self._name, mode)
E OSError: dlopen(/Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/lib_lightgbm.so, 6): no suitable image found. Did find:
E /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/lib_lightgbm.so: mach-o, but wrong architecture
E /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/lib_lightgbm.so: mach-o, but wrong architecture
______________________________________________________________________ ERROR collecting python_package_test/test_utilities.py _______________________________________________________________________
python_package_test/test_utilities.py:6: in <module>
import lightgbm as lgb
../../../opt/anaconda3/lib/python3.8/site-packages/lightgbm/__init__.py:8: in <module>
from .basic import Booster, Dataset, Sequence, register_logger
../../../opt/anaconda3/lib/python3.8/site-packages/lightgbm/basic.py:111: in <module>
_LIB = _load_lib()
../../../opt/anaconda3/lib/python3.8/site-packages/lightgbm/basic.py:102: in _load_lib
lib = ctypes.cdll.LoadLibrary(lib_path[0])
../../../opt/anaconda3/lib/python3.8/ctypes/__init__.py:459: in LoadLibrary
return self._dlltype(name)
../../../opt/anaconda3/lib/python3.8/ctypes/__init__.py:381: in __init__
self._handle = _dlopen(self._name, mode)
E OSError: dlopen(/Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/lib_lightgbm.so, 6): no suitable image found. Did find:
E /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/lib_lightgbm.so: mach-o, but wrong architecture
E /Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/lib_lightgbm.so: mach-o, but wrong architecture
====================================================================================== short test summary info ======================================================================================
ERROR c_api_test/test_.py - OSError: dlopen(/Users/thvasilo/repos/lgbm-master/lib_lightgbm.so, 6): no suitable image found. Did find:
ERROR python_package_test/test_basic.py - OSError: dlopen(/Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/lib_lightgbm.so, 6): no suitable image found. Did find:
ERROR python_package_test/test_consistency.py - OSError: dlopen(/Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/lib_lightgbm.so, 6): no suitable image found. Did find:
ERROR python_package_test/test_dask.py - OSError: dlopen(/Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/lib_lightgbm.so, 6): no suitable image found. Did find:
ERROR python_package_test/test_dual.py - OSError: dlopen(/Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/lib_lightgbm.so, 6): no suitable image found. Did find:
ERROR python_package_test/test_engine.py - OSError: dlopen(/Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/lib_lightgbm.so, 6): no suitable image found. Did find:
ERROR python_package_test/test_plotting.py - OSError: dlopen(/Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/lib_lightgbm.so, 6): no suitable image found. Did find:
ERROR python_package_test/test_sklearn.py - OSError: dlopen(/Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/lib_lightgbm.so, 6): no suitable image found. Did find:
ERROR python_package_test/test_utilities.py - OSError: dlopen(/Users/thvasilo/opt/anaconda3/lib/python3.8/site-packages/lightgbm/lib_lightgbm.so, 6): no suitable image found. Did find:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 9 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
========================================================================================= 9 errors in 6.77s ======================================================================================
Just to clarify: have you tried to run any other Python package that depends on cpp code in your current anaconda3/lib/python3.8
environment? XGBoost, tensorflow, Pytorch, etc.
Could you please try to install Python and all packages via Miniforge
instead of conda
?
https://github.com/conda-forge/miniforge#download
https://stackoverflow.com/a/68228696
If I understood correctly, default conda channels lack support of macOS-arm64 packages. Compare:
Also, please try to play around with CMake's -DCMAKE_OSX_ARCHITECTURES=arm64
flag checking what architecture was actually built by the lipo
command:
https://developer.apple.com/forums/thread/687727
I want to add, I just noticed that CRAN is now testing R packages on macOS with arm64
CPUs.
And {lightgbm}
(the R package for LightGBM) is passing those checks. (build log)
According to https://cran.r-project.org/web/checks/check_flavors.html#r-release-macos-arm64, those checks are run with the following specs:
One big difference between the R package submitted to CRAN and other builds of LightGBM is that it uses the GNU automake to generate Makefiles instead of CMake (see R-package/configure).
So I think that maybe this is additional evidence that LightGBM's C/C++ source code isn't inherently incompatible with the M1 Macs, and that changes to its CMake configuration (as @StrikerRUS suggested) might be required.
So I think that maybe this is additional evidence that LightGBM's C/C++ source code isn't inherently incompatible with the M1 Macs, and that changes to its CMake configuration (as @StrikerRUS suggested) might be required.
Another evidence that LightGBM works fine on M1 is Homebrew: they build some bottles on macOS ARM64 machines:
CPU: octa-core 64-bit arm_firestorm_icestorm
Clang: 13.0.0 build 1300
Git: 2.30.1 => /usr/bin/git
Curl: 7.77.0 => /usr/bin/curl
macOS: 12.0.1-arm64
CLT: 13.1.0.0.1.1633545042
Xcode: 13.1
Rosetta 2: false
https://github.com/Homebrew/homebrew-core/runs/4740548221?check_suite_focus=true
And in contrast to CRAN compilation, Homebrew builds LightGBM with CMake via the following commands: https://github.com/Homebrew/homebrew-core/blob/c34e36ed9feb170d4b10ed72d5bb4cb6f83d2a5f/Formula/lightgbm.rb#L22-L26 https://github.com/Homebrew/brew/blob/069ab087f99fd183aee599bf785cb835d5868407/Library/Homebrew/formula.rb#L1514-L1533
According to this, I think that some changes might be needed in some particular environments, but they are not required by default.
I wish I could provide any better help here, but unfortunately I have no access to ordinary, not to mention M1, macOS machine... 🙁
I strongly believe that the root cause here is Python and LightGBM architectures mismatch.
m1 python 3.7 也是同样的问题
this is bug with m1 python 3.7 ???
this is bug with m1. python 3.7?
@captainIT I guess any version of Python may suffer from architectures mismatch problem.
@captainIT I guess any version of Python may suffer from architectures mismatch problem.
多谢,我m1芯片 改用conda 来创建环境,已经可以正常使用lightGbm了
@captainIT I guess any version of Python may suffer from architectures mismatch problem.
多谢,我m1芯片 改用conda 来创建环境,已经可以正常使用lightGbm了
朋友 可否请教一下安装过程呢,我用conda创建环境还没解决这个问题
Recently, I've had a chance to work on Mac M1 for a few minutes. So I checked whether LightGBM can be installed on it. Good news, it can in both ways: via manual compilation and indirectly triggered compilation after running pip install lightgbm
!
Here are logs of the commands that I used (according to our official installation guide) to install lightgbm package and successfully run it on arm64 architecture.
brew install cmake
brew install libomp
For brew
commands logs are unavailable unfortunately.
(base) leonidalekseev@MacBook-Air-Leonid ~ % git clone --recursive https://github.com/microsoft/LightGBM.git
Cloning into 'LightGBM'...
remote: Enumerating objects: 25115, done.
remote: Counting objects: 100% (791/791), done.
remote: Compressing objects: 100% (434/434), done.
remote: Total 25115 (delta 508), reused 568 (delta 350), pack-reused 24324
Receiving objects: 100% (25115/25115), 18.73 MiB | 691.00 KiB/s, done.
Resolving deltas: 100% (18457/18457), done.
Submodule 'include/boost/compute' (https://github.com/boostorg/compute) registered for path 'external_libs/compute'
Submodule 'eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'external_libs/eigen'
Submodule 'external_libs/fast_double_parser' (https://github.com/lemire/fast_double_parser.git) registered for path 'external_libs/fast_double_parser'
Submodule 'external_libs/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'external_libs/fmt'
Cloning into '/Users/leonidalekseev/LightGBM/external_libs/compute'...
remote: Enumerating objects: 21733, done.
remote: Counting objects: 100% (5/5), done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 21733 (delta 1), reused 2 (delta 0), pack-reused 21728
Receiving objects: 100% (21733/21733), 8.51 MiB | 711.00 KiB/s, done.
Resolving deltas: 100% (17567/17567), done.
Cloning into '/Users/leonidalekseev/LightGBM/external_libs/eigen'...
remote: Enumerating objects: 114927, done.
remote: Counting objects: 100% (91/91), done.
remote: Compressing objects: 100% (73/73), done.
remote: Total 114927 (delta 49), reused 44 (delta 18), pack-reused 114836
Receiving objects: 100% (114927/114927), 102.09 MiB | 1.06 MiB/s, done.
Resolving deltas: 100% (94768/94768), done.
Cloning into '/Users/leonidalekseev/LightGBM/external_libs/fast_double_parser'...
remote: Enumerating objects: 692, done.
remote: Counting objects: 100% (192/192), done.
remote: Compressing objects: 100% (124/124), done.
remote: Total 692 (delta 95), reused 99 (delta 41), pack-reused 500
Receiving objects: 100% (692/692), 802.86 KiB | 769.00 KiB/s, done.
Resolving deltas: 100% (349/349), done.
Cloning into '/Users/leonidalekseev/LightGBM/external_libs/fmt'...
remote: Enumerating objects: 28976, done.
remote: Counting objects: 100% (336/336), done.
remote: Compressing objects: 100% (93/93), done.
remote: Total 28976 (delta 199), reused 306 (delta 192), pack-reused 28640
Receiving objects: 100% (28976/28976), 13.87 MiB | 689.00 KiB/s, done.
Resolving deltas: 100% (19550/19550), done.
Submodule path 'external_libs/compute': checked out '36350b7de849300bd3d72a05d8bf890ca405a014'
Submodule path 'external_libs/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
Submodule path 'external_libs/fast_double_parser': checked out 'ace60646c02dc54c57f19d644e49a61e7e7758ec'
Submodule 'benchmark/dependencies/abseil-cpp' (https://github.com/abseil/abseil-cpp.git) registered for path 'external_libs/fast_double_parser/benchmarks/dependencies/abseil-cpp'
Submodule 'benchmark/dependencies/double-conversion' (https://github.com/google/double-conversion.git) registered for path 'external_libs/fast_double_parser/benchmarks/dependencies/double-conversion'
Cloning into '/Users/leonidalekseev/LightGBM/external_libs/fast_double_parser/benchmarks/dependencies/abseil-cpp'...
remote: Enumerating objects: 16312, done.
remote: Counting objects: 100% (438/438), done.
remote: Compressing objects: 100% (317/317), done.
remote: Total 16312 (delta 251), reused 207 (delta 121), pack-reused 15874
Receiving objects: 100% (16312/16312), 10.64 MiB | 910.00 KiB/s, done.
Resolving deltas: 100% (12531/12531), done.
Cloning into '/Users/leonidalekseev/LightGBM/external_libs/fast_double_parser/benchmarks/dependencies/double-conversion'...
remote: Enumerating objects: 1338, done.
remote: Counting objects: 100% (182/182), done.
remote: Compressing objects: 100% (140/140), done.
remote: Total 1338 (delta 98), reused 85 (delta 35), pack-reused 1156
Receiving objects: 100% (1338/1338), 7.14 MiB | 785.00 KiB/s, done.
Resolving deltas: 100% (870/870), done.
Submodule path 'external_libs/fast_double_parser/benchmarks/dependencies/abseil-cpp': checked out 'd936052d32a5b7ca08b0199a6724724aea432309'
Submodule path 'external_libs/fast_double_parser/benchmarks/dependencies/double-conversion': checked out 'f4cb2384efa55dee0e6652f8674b05763441ab09'
Submodule path 'external_libs/fmt': checked out 'b6f4ceaed0a0a24ccf575fab6c56dd50ccf6f1a9'
(base) leonidalekseev@MacBook-Air-Leonid ~ % cd LightGBM
(base) leonidalekseev@MacBook-Air-Leonid LightGBM % mkdir build
(base) leonidalekseev@MacBook-Air-Leonid LightGBM % cd build
(base) leonidalekseev@MacBook-Air-Leonid build % cmake ..
-- The C compiler identification is AppleClang 13.0.0.13000029
-- The CXX compiler identification is AppleClang 13.0.0.13000029
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found OpenMP_C: -Xclang -fopenmp (found version "5.0")
-- Found OpenMP_CXX: -Xclang -fopenmp (found version "5.0")
-- Found OpenMP: TRUE (found version "5.0")
-- Performing Test MM_PREFETCH
-- Performing Test MM_PREFETCH - Failed
-- Performing Test MM_MALLOC
-- Performing Test MM_MALLOC - Success
-- Using _mm_malloc
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/leonidalekseev/LightGBM/build
(base) leonidalekseev@MacBook-Air-Leonid build % make -j4
[ 5%] Building CXX object CMakeFiles/lightgbm_capi_objs.dir/src/c_api.cpp.o
[ 5%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/boosting/boosting.cpp.o
[ 8%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/boosting/gbdt.cpp.o
[ 10%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/boosting/gbdt_model_text.cpp.o
[ 13%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/boosting/gbdt_prediction.cpp.o
[ 16%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/boosting/prediction_early_stop.cpp.o
[ 18%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/bin.cpp.o
[ 21%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/config.cpp.o
[ 24%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/config_auto.cpp.o
[ 29%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/dataset_loader.cpp.o
[ 29%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/dataset.cpp.o
[ 29%] Built target lightgbm_capi_objs
[ 32%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/file_io.cpp.o
[ 35%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/json11.cpp.o
[ 37%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/metadata.cpp.o
[ 40%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/parser.cpp.o
[ 43%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/train_share_states.cpp.o
[ 45%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/tree.cpp.o
[ 48%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/metric/dcg_calculator.cpp.o
[ 51%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/metric/metric.cpp.o
[ 54%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/network/ifaddrs_patch.cpp.o
[ 56%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/network/linker_topo.cpp.o
[ 59%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/network/linkers_mpi.cpp.o
[ 62%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/network/linkers_socket.cpp.o
[ 64%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/network/network.cpp.o
[ 67%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/objective/objective_function.cpp.o
[ 70%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/cuda_tree_learner.cpp.o
[ 72%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/data_parallel_tree_learner.cpp.o
[ 75%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/feature_parallel_tree_learner.cpp.o
[ 78%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/gpu_tree_learner.cpp.o
[ 81%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/linear_tree_learner.cpp.o
[ 83%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/serial_tree_learner.cpp.o
[ 86%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/tree_learner.cpp.o
[ 89%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/voting_parallel_tree_learner.cpp.o
[ 89%] Built target lightgbm_objs
[ 97%] Building CXX object CMakeFiles/lightgbm.dir/src/application/application.cpp.o
[ 97%] Linking CXX shared library ../lib_lightgbm.so
[ 97%] Building CXX object CMakeFiles/lightgbm.dir/src/main.cpp.o
[ 97%] Built target _lightgbm
[100%] Linking CXX executable ../lightgbm
[100%] Built target lightgbm
(base) leonidalekseev@MacBook-Air-Leonid build % cd ../python-package
(base) leonidalekseev@MacBook-Air-Leonid python-package % python setup.py install --precompile
running install
running build
running build_py
creating build
creating build/lib
creating build/lib/lightgbm
copying lightgbm/callback.py -> build/lib/lightgbm
copying lightgbm/compat.py -> build/lib/lightgbm
copying lightgbm/plotting.py -> build/lib/lightgbm
copying lightgbm/__init__.py -> build/lib/lightgbm
copying lightgbm/engine.py -> build/lib/lightgbm
copying lightgbm/dask.py -> build/lib/lightgbm
copying lightgbm/basic.py -> build/lib/lightgbm
copying lightgbm/libpath.py -> build/lib/lightgbm
copying lightgbm/sklearn.py -> build/lib/lightgbm
running egg_info
creating lightgbm.egg-info
writing lightgbm.egg-info/PKG-INFO
writing dependency_links to lightgbm.egg-info/dependency_links.txt
writing requirements to lightgbm.egg-info/requires.txt
writing top-level names to lightgbm.egg-info/top_level.txt
writing manifest file 'lightgbm.egg-info/SOURCES.txt'
reading manifest file 'lightgbm.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
no previously-included directories found matching 'build'
warning: no files found matching 'LICENSE'
warning: no files found matching '*.txt'
warning: no files found matching '*.so' under directory 'lightgbm'
warning: no files found matching 'compile/CMakeLists.txt'
warning: no files found matching 'compile/cmake/IntegratedOpenCL.cmake'
warning: no files found matching '*.so' under directory 'compile'
warning: no files found matching '*.dll' under directory 'compile/Release'
warning: no files found matching 'compile/external_libs/compute/CMakeLists.txt'
warning: no files found matching '*' under directory 'compile/external_libs/compute/cmake'
warning: no files found matching '*' under directory 'compile/external_libs/compute/include'
warning: no files found matching '*' under directory 'compile/external_libs/compute/meta'
warning: no files found matching 'compile/external_libs/eigen/CMakeLists.txt'
warning: no files found matching 'compile/external_libs/eigen/Eigen/Cholesky'
warning: no files found matching 'compile/external_libs/eigen/Eigen/Core'
warning: no files found matching 'compile/external_libs/eigen/Eigen/Dense'
warning: no files found matching 'compile/external_libs/eigen/Eigen/Eigenvalues'
warning: no files found matching 'compile/external_libs/eigen/Eigen/Geometry'
warning: no files found matching 'compile/external_libs/eigen/Eigen/Householder'
warning: no files found matching 'compile/external_libs/eigen/Eigen/Jacobi'
warning: no files found matching 'compile/external_libs/eigen/Eigen/LU'
warning: no files found matching 'compile/external_libs/eigen/Eigen/QR'
warning: no files found matching 'compile/external_libs/eigen/Eigen/SVD'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/Cholesky'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/Core'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/Eigenvalues'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/Geometry'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/Householder'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/Jacobi'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/LU'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/misc'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/plugins'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/QR'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/SVD'
warning: no files found matching 'compile/external_libs/fast_double_parser/CMakeLists.txt'
warning: no files found matching 'compile/external_libs/fast_double_parser/LICENSE'
warning: no files found matching 'compile/external_libs/fast_double_parser/LICENSE.BSL'
warning: no files found matching '*' under directory 'compile/external_libs/fast_double_parser/include'
warning: no files found matching 'compile/external_libs/fmt/CMakeLists.txt'
warning: no files found matching 'compile/external_libs/fmt/LICENSE.rst'
warning: no files found matching '*' under directory 'compile/external_libs/fmt/include'
warning: no files found matching '*' under directory 'compile/include'
warning: no files found matching '*' under directory 'compile/src'
warning: no files found matching 'LightGBM.sln' under directory 'compile/windows'
warning: no files found matching 'LightGBM.vcxproj' under directory 'compile/windows'
warning: no files found matching '*.dll' under directory 'compile/windows/x64/DLL'
warning: no previously-included files matching '*.py[co]' found anywhere in distribution
warning: no previously-included files found matching 'compile/external_libs/compute/.git'
writing manifest file 'lightgbm.egg-info/SOURCES.txt'
copying lightgbm/VERSION.txt -> build/lib/lightgbm
running install_lib
creating /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages/lightgbm
copying build/lib/lightgbm/callback.py -> /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages/lightgbm
copying build/lib/lightgbm/compat.py -> /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages/lightgbm
copying build/lib/lightgbm/plotting.py -> /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages/lightgbm
copying build/lib/lightgbm/__init__.py -> /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages/lightgbm
copying build/lib/lightgbm/engine.py -> /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages/lightgbm
copying build/lib/lightgbm/dask.py -> /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages/lightgbm
copying build/lib/lightgbm/basic.py -> /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages/lightgbm
copying build/lib/lightgbm/libpath.py -> /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages/lightgbm
copying build/lib/lightgbm/sklearn.py -> /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages/lightgbm
copying build/lib/lightgbm/VERSION.txt -> /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages/lightgbm
INFO:LightGBM:Installing lib_lightgbm from: ['/Users/leonidalekseev/LightGBM/lib_lightgbm.so']
copying /Users/leonidalekseev/LightGBM/lib_lightgbm.so -> /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages/lightgbm
byte-compiling /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages/lightgbm/callback.py to callback.cpython-39.pyc
byte-compiling /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages/lightgbm/compat.py to compat.cpython-39.pyc
byte-compiling /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages/lightgbm/plotting.py to plotting.cpython-39.pyc
byte-compiling /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages/lightgbm/__init__.py to __init__.cpython-39.pyc
byte-compiling /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages/lightgbm/engine.py to engine.cpython-39.pyc
byte-compiling /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages/lightgbm/dask.py to dask.cpython-39.pyc
byte-compiling /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages/lightgbm/basic.py to basic.cpython-39.pyc
byte-compiling /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages/lightgbm/libpath.py to libpath.cpython-39.pyc
byte-compiling /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages/lightgbm/sklearn.py to sklearn.cpython-39.pyc
running install_egg_info
Copying lightgbm.egg-info to /opt/homebrew/Caskroom/miniforge/base/lib/python3.9/site-packages/lightgbm-3.3.2.99-py3.9.egg-info
running install_scripts
(base) leonidalekseev@MacBook-Air-Leonid python-package % python
Python 3.9.6 | packaged by conda-forge | (default, Jul 11 2021, 03:35:11)
[Clang 11.1.0 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import lightgbm as lgb
>>> import numpy as np
>>> X = np.random.random((50, 10))
>>> y = np.random.random(50)
>>> lgb.LGBMRegressor().fit(X, y)
[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.000235 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 180
[LightGBM] [Info] Number of data points in the train set: 50, number of used features: 10
[LightGBM] [Info] Start training from score 0.486477
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
LGBMRegressor()
As I suggested earlier in this thread, miniforge arm64 version was used to run this experiment.
leonidalekseev@MacBook-Air-Leonid ~ % conda info
active environment : None
shell level : 0
user config file : /Users/leonidalekseev/.condarc
populated config files : /opt/homebrew/Caskroom/miniforge/base/.condarc
/Users/leonidalekseev/.condarc
conda version : 4.10.3
conda-build version : not installed
python version : 3.9.6.final.0
virtual packages : __osx=12.0.1=0
__unix=0=0
__archspec=1=arm64
base environment : /opt/homebrew/Caskroom/miniforge/base (writable)
conda av data dir : /opt/homebrew/Caskroom/miniforge/base/etc/conda
conda av metadata url : None
channel URLs : https://conda.anaconda.org/conda-forge/osx-arm64
https://conda.anaconda.org/conda-forge/noarch
package cache : /opt/homebrew/Caskroom/miniforge/base/pkgs
/Users/leonidalekseev/.conda/pkgs
envs directories : /opt/homebrew/Caskroom/miniforge/base/envs
/Users/leonidalekseev/.conda/envs
platform : osx-arm64
user-agent : conda/4.10.3 requests/2.26.0 CPython/3.9.6 Darwin/21.1.0 OSX/12.0.1
UID:GID : 501:20
netrc file : None
offline mode : False
I tried the whole clone/library-build/python-build/train sequence on a new Conda env but still get the same error as before:
$ git clone git@github.com:microsoft/LightGBM.git lgbm-m1-test --recursive
$ cd lgbm-m1-test
$ mkdir build && cd build
(base) build $ cmake ..
-- The C compiler identification is AppleClang 13.0.0.13000029
-- The CXX compiler identification is AppleClang 13.0.0.13000029
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found OpenMP_C: -Xclang -fopenmp (found version "5.0")
-- Found OpenMP_CXX: -Xclang -fopenmp (found version "5.0")
-- Found OpenMP: TRUE (found version "5.0")
-- Performing Test MM_PREFETCH
-- Performing Test MM_PREFETCH - Failed
-- Performing Test MM_MALLOC
-- Performing Test MM_MALLOC - Success
-- Using _mm_malloc
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/thvasilo/repos/lgbm-m1-test/build
(base) build $ make -j8
[ 8%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/boosting/gbdt.cpp.o
[ 13%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/boosting/gbdt_model_text.cpp.o
[ 13%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/boosting/prediction_early_stop.cpp.o
[ 13%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/boosting/gbdt_prediction.cpp.o
[ 13%] Building CXX object CMakeFiles/lightgbm_capi_objs.dir/src/c_api.cpp.o
[ 16%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/boosting/boosting.cpp.o
[ 18%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/config.cpp.o
[ 21%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/bin.cpp.o
[ 24%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/config_auto.cpp.o
[ 27%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/dataset.cpp.o
[ 29%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/dataset_loader.cpp.o
[ 32%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/file_io.cpp.o
[ 35%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/json11.cpp.o
[ 37%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/metadata.cpp.o
[ 40%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/parser.cpp.o
[ 43%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/train_share_states.cpp.o
[ 45%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/io/tree.cpp.o
[ 48%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/metric/dcg_calculator.cpp.o
[ 51%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/metric/metric.cpp.o
[ 54%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/network/ifaddrs_patch.cpp.o
[ 56%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/network/linker_topo.cpp.o
[ 59%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/network/linkers_mpi.cpp.o
[ 62%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/network/linkers_socket.cpp.o
[ 64%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/network/network.cpp.o
[ 67%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/objective/objective_function.cpp.o
[ 70%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/cuda_tree_learner.cpp.o
[ 72%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/data_parallel_tree_learner.cpp.o
[ 75%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/feature_parallel_tree_learner.cpp.o
[ 78%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/gpu_tree_learner.cpp.o
[ 81%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/linear_tree_learner.cpp.o
[ 83%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/serial_tree_learner.cpp.o
[ 83%] Built target lightgbm_capi_objs
[ 86%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/tree_learner.cpp.o
[ 89%] Building CXX object CMakeFiles/lightgbm_objs.dir/src/treelearner/voting_parallel_tree_learner.cpp.o
[ 89%] Built target lightgbm_objs
[ 91%] Building CXX object CMakeFiles/lightgbm.dir/src/application/application.cpp.o
[ 97%] Building CXX object CMakeFiles/lightgbm.dir/src/main.cpp.o
[ 97%] Linking CXX shared library ../lib_lightgbm.so
[ 97%] Built target _lightgbm
[100%] Linking CXX executable ../lightgbm
[100%] Built target lightgbm
(base) build $ cd ../python-package/
(base) python-package $ conda create --name lgbm-m1 scipy numpy
Collecting package metadata (current_repodata.json): done
Solving environment: done
## Package Plan ##
environment location: /Users/thvasilo/opt/anaconda3/envs/lgbm-m1
added / updated specs:
- numpy
- scipy
The following packages will be downloaded:
package | build
---------------------------|-----------------
ca-certificates-2022.2.1 | hecd8cb5_0 122 KB
six-1.16.0 | pyhd3eb1b0_1 18 KB
sqlite-3.37.2 | h707629a_0 1.2 MB
------------------------------------------------------------
Total: 1.3 MB
The following NEW packages will be INSTALLED:
blas pkgs/main/osx-64::blas-1.0-mkl
ca-certificates pkgs/main/osx-64::ca-certificates-2022.2.1-hecd8cb5_0
certifi pkgs/main/osx-64::certifi-2021.10.8-py39hecd8cb5_2
intel-openmp pkgs/main/osx-64::intel-openmp-2021.4.0-hecd8cb5_3538
libcxx pkgs/main/osx-64::libcxx-12.0.0-h2f01273_0
libffi pkgs/main/osx-64::libffi-3.3-hb1e8313_2
libgfortran pkgs/main/osx-64::libgfortran-3.0.1-h93005f0_2
mkl pkgs/main/osx-64::mkl-2021.4.0-hecd8cb5_637
mkl-service pkgs/main/osx-64::mkl-service-2.4.0-py39h9ed2024_0
mkl_fft pkgs/main/osx-64::mkl_fft-1.3.1-py39h4ab4a9b_0
mkl_random pkgs/main/osx-64::mkl_random-1.2.2-py39hb2f4e1b_0
ncurses pkgs/main/osx-64::ncurses-6.3-hca72f7f_2
numpy pkgs/main/osx-64::numpy-1.21.2-py39h4b4dc7a_0
numpy-base pkgs/main/osx-64::numpy-base-1.21.2-py39he0bd621_0
openssl pkgs/main/osx-64::openssl-1.1.1m-hca72f7f_0
pip pkgs/main/osx-64::pip-21.2.4-py39hecd8cb5_0
python pkgs/main/osx-64::python-3.9.7-h88f2d9e_1
readline pkgs/main/osx-64::readline-8.1.2-hca72f7f_1
scipy pkgs/main/osx-64::scipy-1.7.3-py39h8c7af03_0
setuptools pkgs/main/osx-64::setuptools-58.0.4-py39hecd8cb5_0
six pkgs/main/noarch::six-1.16.0-pyhd3eb1b0_1
sqlite pkgs/main/osx-64::sqlite-3.37.2-h707629a_0
tk pkgs/main/osx-64::tk-8.6.11-h7bc2e8c_0
tzdata pkgs/main/noarch::tzdata-2021e-hda174b7_0
wheel pkgs/main/noarch::wheel-0.37.1-pyhd3eb1b0_0
xz pkgs/main/osx-64::xz-5.2.5-h1de35cc_0
zlib pkgs/main/osx-64::zlib-1.2.11-h4dc903c_4
Proceed ([y]/n)? y
Downloading and Extracting Packages
sqlite-3.37.2 | 1.2 MB | ###################################################################################################################################### | 100%
six-1.16.0 | 18 KB | ###################################################################################################################################### | 100%
ca-certificates-2022 | 122 KB | ###################################################################################################################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
# $ conda activate lgbm-m1
#
# To deactivate an active environment, use
#
# $ conda deactivate
(base) python-package $ conda activate lgbm-m1
(lgbm-m1) python-package $ python setup.py install --precompile
running install
running build
running build_py
creating build
creating build/lib
creating build/lib/lightgbm
copying lightgbm/callback.py -> build/lib/lightgbm
copying lightgbm/compat.py -> build/lib/lightgbm
copying lightgbm/plotting.py -> build/lib/lightgbm
copying lightgbm/__init__.py -> build/lib/lightgbm
copying lightgbm/engine.py -> build/lib/lightgbm
copying lightgbm/dask.py -> build/lib/lightgbm
copying lightgbm/basic.py -> build/lib/lightgbm
copying lightgbm/libpath.py -> build/lib/lightgbm
copying lightgbm/sklearn.py -> build/lib/lightgbm
running egg_info
creating lightgbm.egg-info
writing lightgbm.egg-info/PKG-INFO
writing dependency_links to lightgbm.egg-info/dependency_links.txt
writing requirements to lightgbm.egg-info/requires.txt
writing top-level names to lightgbm.egg-info/top_level.txt
writing manifest file 'lightgbm.egg-info/SOURCES.txt'
reading manifest file 'lightgbm.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
no previously-included directories found matching 'build'
warning: no files found matching 'LICENSE'
warning: no files found matching '*.txt'
warning: no files found matching '*.so' under directory 'lightgbm'
warning: no files found matching 'compile/CMakeLists.txt'
warning: no files found matching 'compile/cmake/IntegratedOpenCL.cmake'
warning: no files found matching '*.so' under directory 'compile'
warning: no files found matching '*.dll' under directory 'compile/Release'
warning: no files found matching 'compile/external_libs/compute/CMakeLists.txt'
warning: no files found matching '*' under directory 'compile/external_libs/compute/cmake'
warning: no files found matching '*' under directory 'compile/external_libs/compute/include'
warning: no files found matching '*' under directory 'compile/external_libs/compute/meta'
warning: no files found matching 'compile/external_libs/eigen/CMakeLists.txt'
warning: no files found matching 'compile/external_libs/eigen/Eigen/Cholesky'
warning: no files found matching 'compile/external_libs/eigen/Eigen/Core'
warning: no files found matching 'compile/external_libs/eigen/Eigen/Dense'
warning: no files found matching 'compile/external_libs/eigen/Eigen/Eigenvalues'
warning: no files found matching 'compile/external_libs/eigen/Eigen/Geometry'
warning: no files found matching 'compile/external_libs/eigen/Eigen/Householder'
warning: no files found matching 'compile/external_libs/eigen/Eigen/Jacobi'
warning: no files found matching 'compile/external_libs/eigen/Eigen/LU'
warning: no files found matching 'compile/external_libs/eigen/Eigen/QR'
warning: no files found matching 'compile/external_libs/eigen/Eigen/SVD'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/Cholesky'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/Core'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/Eigenvalues'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/Geometry'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/Householder'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/Jacobi'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/LU'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/misc'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/plugins'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/QR'
warning: no files found matching '*' under directory 'compile/external_libs/eigen/Eigen/src/SVD'
warning: no files found matching 'compile/external_libs/fast_double_parser/CMakeLists.txt'
warning: no files found matching 'compile/external_libs/fast_double_parser/LICENSE'
warning: no files found matching 'compile/external_libs/fast_double_parser/LICENSE.BSL'
warning: no files found matching '*' under directory 'compile/external_libs/fast_double_parser/include'
warning: no files found matching 'compile/external_libs/fmt/CMakeLists.txt'
warning: no files found matching 'compile/external_libs/fmt/LICENSE.rst'
warning: no files found matching '*' under directory 'compile/external_libs/fmt/include'
warning: no files found matching '*' under directory 'compile/include'
warning: no files found matching '*' under directory 'compile/src'
warning: no files found matching 'LightGBM.sln' under directory 'compile/windows'
warning: no files found matching 'LightGBM.vcxproj' under directory 'compile/windows'
warning: no files found matching '*.dll' under directory 'compile/windows/x64/DLL'
warning: no previously-included files matching '*.py[co]' found anywhere in distribution
warning: no previously-included files found matching 'compile/external_libs/compute/.git'
writing manifest file 'lightgbm.egg-info/SOURCES.txt'
copying lightgbm/VERSION.txt -> build/lib/lightgbm
running install_lib
creating /Users/thvasilo/opt/anaconda3/envs/lgbm-m1/lib/python3.9/site-packages/lightgbm
copying build/lib/lightgbm/callback.py -> /Users/thvasilo/opt/anaconda3/envs/lgbm-m1/lib/python3.9/site-packages/lightgbm
copying build/lib/lightgbm/compat.py -> /Users/thvasilo/opt/anaconda3/envs/lgbm-m1/lib/python3.9/site-packages/lightgbm
copying build/lib/lightgbm/plotting.py -> /Users/thvasilo/opt/anaconda3/envs/lgbm-m1/lib/python3.9/site-packages/lightgbm
copying build/lib/lightgbm/__init__.py -> /Users/thvasilo/opt/anaconda3/envs/lgbm-m1/lib/python3.9/site-packages/lightgbm
copying build/lib/lightgbm/engine.py -> /Users/thvasilo/opt/anaconda3/envs/lgbm-m1/lib/python3.9/site-packages/lightgbm
copying build/lib/lightgbm/dask.py -> /Users/thvasilo/opt/anaconda3/envs/lgbm-m1/lib/python3.9/site-packages/lightgbm
copying build/lib/lightgbm/basic.py -> /Users/thvasilo/opt/anaconda3/envs/lgbm-m1/lib/python3.9/site-packages/lightgbm
copying build/lib/lightgbm/libpath.py -> /Users/thvasilo/opt/anaconda3/envs/lgbm-m1/lib/python3.9/site-packages/lightgbm
copying build/lib/lightgbm/sklearn.py -> /Users/thvasilo/opt/anaconda3/envs/lgbm-m1/lib/python3.9/site-packages/lightgbm
copying build/lib/lightgbm/VERSION.txt -> /Users/thvasilo/opt/anaconda3/envs/lgbm-m1/lib/python3.9/site-packages/lightgbm
INFO:LightGBM:Installing lib_lightgbm from: ['/Users/thvasilo/repos/lgbm-m1-test/lib_lightgbm.so']
copying /Users/thvasilo/repos/lgbm-m1-test/lib_lightgbm.so -> /Users/thvasilo/opt/anaconda3/envs/lgbm-m1/lib/python3.9/site-packages/lightgbm
byte-compiling /Users/thvasilo/opt/anaconda3/envs/lgbm-m1/lib/python3.9/site-packages/lightgbm/callback.py to callback.cpython-39.pyc
byte-compiling /Users/thvasilo/opt/anaconda3/envs/lgbm-m1/lib/python3.9/site-packages/lightgbm/compat.py to compat.cpython-39.pyc
byte-compiling /Users/thvasilo/opt/anaconda3/envs/lgbm-m1/lib/python3.9/site-packages/lightgbm/plotting.py to plotting.cpython-39.pyc
byte-compiling /Users/thvasilo/opt/anaconda3/envs/lgbm-m1/lib/python3.9/site-packages/lightgbm/__init__.py to __init__.cpython-39.pyc
byte-compiling /Users/thvasilo/opt/anaconda3/envs/lgbm-m1/lib/python3.9/site-packages/lightgbm/engine.py to engine.cpython-39.pyc
byte-compiling /Users/thvasilo/opt/anaconda3/envs/lgbm-m1/lib/python3.9/site-packages/lightgbm/dask.py to dask.cpython-39.pyc
byte-compiling /Users/thvasilo/opt/anaconda3/envs/lgbm-m1/lib/python3.9/site-packages/lightgbm/basic.py to basic.cpython-39.pyc
byte-compiling /Users/thvasilo/opt/anaconda3/envs/lgbm-m1/lib/python3.9/site-packages/lightgbm/libpath.py to libpath.cpython-39.pyc
byte-compiling /Users/thvasilo/opt/anaconda3/envs/lgbm-m1/lib/python3.9/site-packages/lightgbm/sklearn.py to sklearn.cpython-39.pyc
running install_egg_info
Copying lightgbm.egg-info to /Users/thvasilo/opt/anaconda3/envs/lgbm-m1/lib/python3.9/site-packages/lightgbm-3.3.2.99-py3.9.egg-info
running install_scripts
(lgbm-m1) python-package $ python
Python 3.9.7 (default, Sep 16 2021, 08:50:36)
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import lightgbm as lgb
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/thvasilo/repos/lgbm-m1-test/python-package/lightgbm/__init__.py", line 8, in <module>
from .basic import Booster, Dataset, Sequence, register_logger
File "/Users/thvasilo/repos/lgbm-m1-test/python-package/lightgbm/basic.py", line 110, in <module>
_LIB = _load_lib()
File "/Users/thvasilo/repos/lgbm-m1-test/python-package/lightgbm/basic.py", line 101, in _load_lib
lib = ctypes.cdll.LoadLibrary(lib_path[0])
File "/Users/thvasilo/opt/anaconda3/envs/lgbm-m1/lib/python3.9/ctypes/__init__.py", line 460, in LoadLibrary
return self._dlltype(name)
File "/Users/thvasilo/opt/anaconda3/envs/lgbm-m1/lib/python3.9/ctypes/__init__.py", line 382, in __init__
self._handle = _dlopen(self._name, mode)
OSError: dlopen(/Users/thvasilo/repos/lgbm-m1-test/lib_lightgbm.so, 6): no suitable image found. Did find:
/Users/thvasilo/repos/lgbm-m1-test/lib_lightgbm.so: mach-o, but wrong architecture
/Users/thvasilo/repos/lgbm-m1-test/lib_lightgbm.so: mach-o, but wrong architecture
One difference could be that I'm using Conda instead of miniconda?
conda info
active environment : lgbm-m1
active env location : /Users/thvasilo/opt/anaconda3/envs/lgbm-m1
shell level : 2
user config file : /Users/thvasilo/.condarc
populated config files :
conda version : 4.11.0
conda-build version : 3.21.4
python version : 3.8.8.final.0
virtual packages : __osx=10.16=0
__unix=0=0
__archspec=1=x86_64
base environment : /Users/thvasilo/opt/anaconda3 (writable)
conda av data dir : /Users/thvasilo/opt/anaconda3/etc/conda
conda av metadata url : None
channel URLs : https://repo.anaconda.com/pkgs/main/osx-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/osx-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /Users/thvasilo/opt/anaconda3/pkgs
/Users/thvasilo/.conda/pkgs
envs directories : /Users/thvasilo/opt/anaconda3/envs
/Users/thvasilo/.conda/envs
platform : osx-64
user-agent : conda/4.11.0 requests/2.25.1 CPython/3.8.8 Darwin/20.6.0 OSX/10.16
UID:GID : 505:20
netrc file : None
offline mode : False
One difference ...
... and the second one is macOS version. Yours is 11
(or 10.16
?) and mine was 12
.
One difference could be that I'm using Conda instead of miniconda?
Yeah! That is my main assumption! I believe that LightGBM is being compiled for arm64 architecture but conda (and Python) is x64-based and is run via Rosetta 2 translation. So architecture mismatch happens.
...
virtual packages : __osx=10.16=0
__unix=0=0
__archspec=1=x86_64
...
platform : osx-64
Please, install miniforge instead of conda because it has a native support for arm64 architecture. https://github.com/microsoft/LightGBM/issues/4843#issuecomment-1012313487
BTW, are you able to run XGBoost for example with your current setup?
When will M1-compatible wheels be published to pypi?
@freddyaboulton I guess right after a free CI service with M1 will be appeared. Related: https://github.com/actions/virtual-environments/issues/2187.
+1 to @StrikerRUS 's comment. For anyone finding this thread from search engines, it will be very difficult for LightGBM to confidently provide support for M1 Macs without access to free CI resources using M1 Macs.
@jameslamb What about using cibuildwheel
? I don't have first-hand experience but it seems there are open source data science projects using it to build wheels for osx-arm with github actions:
https://github.com/PyWavelets/pywt/blob/master/.github/workflows/wheel_tests_and_release.yml
what about using
cibuildwheel
?
Maybe! We have a separate discussion going on in #5061 about possibly changing the toolchain for how this project builds wheels.
But the most important thing we need to be able to provide a wheel for the M1 Mac is a free CI resource, not different build tooling. Without that, it won't be possible to guarantee the changes to the codebase don't break compatibility with the M1 Macs.
@jameslamb In that github action I linked, wheels are built for M1 Mac on macos-latest
using cibuildwheel
. So looks like the free CI resource is already there if you use cibuildwheel
.
I think that the runners for macos-latest
still use Intel processors. I believe the job you've linked using cibuildwheel
is using cross-compilation, not truly compiling on + testing on an M1 Mac with an ARM chip.
CIBW_ARCHS_MACOS: x86_64 arm64 universal2
From https://cibuildwheel.readthedocs.io/en/stable/options/#archs
CIBW_ARCHS Change the architectures built on your machine by default. A list of architectures to build. On macOS, this option can be used to cross-compile between x86_64, universal2 and arm64 for Apple Silicon support
The jobs there look super useful, so thanks again for sharing them! But ideally we'd like to test building from source on an actual Mac with an ARM chip.
That's why we're closely watching https://github.com/actions/virtual-environments/issues/2187.
And I found another discussion tonight where GitHub indicated that they're actively working on adding an ARM MacOS runner image: https://github.com/actions/runner/issues/805#issuecomment-1041968059.
This issue has been automatically closed because it has been awaiting a response for too long. When you have time to to work with the maintainers to resolve this issue, please post a new comment and it will be re-opened. If the issue has been locked for editing by the time you return to it, please open a new issue and reference this one. Thank you for taking the time to improve LightGBM!
Adding that there is now a formal item on GitHub's public roadmap for GitHub actions, tracking support for using M1 Macs in GitHub-hosted runners: https://github.com/github/roadmap/issues/528
I strongly believe that @thvasilo is right and this is
a common problem with other Python packages and M1 Macs
and we can't do anything with it. Probably this affects Python 3.9 version only.
Some random examples:
- https://stackoverflow.com/questions/67983582/python-oserrordlopen-no-suitable-image-found-mac-m1
- Mac M1 support pyusb/pyusb#355
- https://stackoverflow.com/questions/68561453/m1-mac-gdal-wrong-architecture-error-django
- https://stackoverflow.com/questions/70207714/problem-importing-camb-python-package-on-m1-mac
- https://issueexplorer.com/issue/kitao/pyxel/299
@thvasilo Also, raw logs may help. Please install LightGBM in the following way:
brew install cmake brew install libomp git clone --recursive https://github.com/microsoft/LightGBM cd LightGBM mkdir build cd build cmake .. make -j4
according to https://lightgbm.readthedocs.io/en/latest/Installation-Guide.html#build-from-github and after that run
cd ../python-package python setup.py install --precompile
I did exactly this and it worked seemlessly for me on my m1 pro 14 inch macbook.
This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.
Description
I'm trying to do some development on my M1 Mac, and while I'm able to generate the libraries and executables as expected, when try to run the Python tests I get the following error:
Reproducible example
Environment info
LightGBM version or commit hash:
commit 8f4126d66065347519185b2f687d423117d67b1b
Command(s) you used to install LightGBM
python setup.py install
conda list:
Additional Comments
This seems to be a common problem with other Python packages and M1 Macs, but I haven't been able to find a consistent solution.