search

wannesm / dtaidistance

Time series distances: Dynamic Time Warping (fast DTW implementation in C)
Other
1.08k stars 184 forks source link

Apple M1 Pro: Failed to import C version of dtaidistance #180

Closed tommedema closed 1 year ago

tommedema commented 1 year ago

I just changed devices to a Macbook with Apple M1 Pro (arm) chipset and unfortunately cannot get dtaidistance to run with the C version.

I've tried the following which used to work on my intel macbook:

pip install --force-reinstall --no-deps --no-build-isolation --no-binary dtaidistance git+https://github.com/wannesm/dtaidistance.git#egg=dtaidistance

When I run:

from dtaidistance import dtw
dtw.try_import_c(verbose = True)

I get:

Cannot import OMP-based library (dtw_cc_omp)
Cannot import Numpy-based library (dtw_cc_numpy)

Not all libraries are available in your installation. 
You can rerun the compilation from source or pip install in verbose mode:
pip install -vvv --upgrade --force-reinstall --no-deps --no-binary dtaidistance dtaidistance
In case you need to use an older version of numpy, compile against your current installation:
pip install -vvv --upgrade --force-reinstall --no-deps --no-build-isolation --no-binary dtaidistance dtaidistance

Share the following information when submitting a bug report:
== Packages ==
- Cannot import OMP-based library (dtw_cc_omp)
- cannot import name 'dtw_cc_omp' from 'dtaidistance' (/Users/playground/.pyenv/versions/3.10.6/lib/python3.10/site-packages/dtaidistance/__init__.py)
- Cannot import Numpy-based library (dtw_cc_numpy)
- cannot import name 'dtw_cc_numpy' from 'dtaidistance' (/Users/playground/.pyenv/versions/3.10.6/lib/python3.10/site-packages/dtaidistance/__init__.py)
- Numpy version: 1.23.3
- Matplotlib version: 3.6.0
- Scipy version: 1.9.1
== System information ==
namespace(name='cpython', cache_tag='cpython-310', version=sys.version_info(major=3, minor=10, micro=6, releaselevel='final', serial=0), hexversion=50988784, _multiarch='darwin')
== Compilation information ==
Could not read compilation.log
[Errno 2] No such file or directory: '/Users/playground/.pyenv/versions/3.10.6/lib/python3.10/site-packages/dtaidistance/compilation.log'

==
False

Other variants I've tried:

pip install --force-reinstall git+https://github.com/wannesm/dtaidistance.git#egg=dtaidistance

pip install --force-reinstall --upgrade --no-deps --no-build-isolation git+https://github.com/wannesm/dtaidistance.git#egg=dtaidistance

pip install --force-reinstall --upgrade --no-deps --no-binary dtaidistance git+https://github.com/wannesm/dtaidistance.git#egg=dtaidistance

pip install --force-reinstall --no-build-isolation --no-binary dtaidistance git+https://github.com/wannesm/dtaidistance.git#egg=dtaidistance

pip install --global-option=--noopenmp --force-reinstall --no-deps --no-build-isolation --no-binary dtaidistance git+https://github.com/wannesm/dtaidistance.git#egg=dtaidistance

pip install --global-option=--noxpreprocessor --force-reinstall --no-deps --no-build-isolation --no-binary dtaidistance git+https://github.com/wannesm/dtaidistance.git#egg=dtaidistance

brew install gcc pip install --global-option=--forcegnugcc --force-reinstall --no-deps --no-build-isolation --no-binary dtaidistance git+https://github.com/wannesm/dtaidistance.git#egg=dtaidistance

When I try to install from source with:

python3 setup.py build_ext --inplace

I get:

`dtaidistance/dtw_cc.c:781:10: fatal error: 'omp.h' file not found

include 

     ^~~~~~~

1 error generated. ERROR: The C extension could not be compiled error: command '/usr/bin/clang' failed with exit code 1 running build_ext Installed the plain Python version of the package. If you need the C extension, try reinstalling.`

I tried running brew install libomp but no success

I also tried python3 setup.py --forcellvm build_ext --inplace and python3 setup.py --forcegnugcc build_ext --inplace

This might also be relevant output:

dtaidistance-master % gcc -v
Apple clang version 14.0.0 (clang-1400.0.29.102)
Target: arm64-apple-darwin21.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

And:

% brew ls gcc
/opt/homebrew/Cellar/gcc/12.2.0/bin/aarch64-apple-darwin21-c++-12
/opt/homebrew/Cellar/gcc/12.2.0/bin/aarch64-apple-darwin21-g++-12
/opt/homebrew/Cellar/gcc/12.2.0/bin/aarch64-apple-darwin21-gcc-12
/opt/homebrew/Cellar/gcc/12.2.0/bin/aarch64-apple-darwin21-gcc-ar-12
/opt/homebrew/Cellar/gcc/12.2.0/bin/aarch64-apple-darwin21-gcc-nm-12
/opt/homebrew/Cellar/gcc/12.2.0/bin/aarch64-apple-darwin21-gcc-ranlib-12
/opt/homebrew/Cellar/gcc/12.2.0/bin/aarch64-apple-darwin21-gfortran-12
/opt/homebrew/Cellar/gcc/12.2.0/bin/c++-12
/opt/homebrew/Cellar/gcc/12.2.0/bin/cpp-12
/opt/homebrew/Cellar/gcc/12.2.0/bin/g++-12
/opt/homebrew/Cellar/gcc/12.2.0/bin/gcc-12
/opt/homebrew/Cellar/gcc/12.2.0/bin/gcc-ar-12
/opt/homebrew/Cellar/gcc/12.2.0/bin/gcc-nm-12
/opt/homebrew/Cellar/gcc/12.2.0/bin/gcc-ranlib-12
/opt/homebrew/Cellar/gcc/12.2.0/bin/gcov-12
/opt/homebrew/Cellar/gcc/12.2.0/bin/gcov-dump-12
/opt/homebrew/Cellar/gcc/12.2.0/bin/gcov-tool-12
/opt/homebrew/Cellar/gcc/12.2.0/bin/gfortran
/opt/homebrew/Cellar/gcc/12.2.0/bin/gfortran-12
/opt/homebrew/Cellar/gcc/12.2.0/bin/lto-dump-12
/opt/homebrew/Cellar/gcc/12.2.0/include/c++/ (806 files)
/opt/homebrew/Cellar/gcc/12.2.0/lib/gcc/ (607 files)
/opt/homebrew/Cellar/gcc/12.2.0/libexec/gcc/ (14 files)
/opt/homebrew/Cellar/gcc/12.2.0/share/gcc-12/ (4 files)
/opt/homebrew/Cellar/gcc/12.2.0/share/man/ (11 files)
wannesm commented 1 year ago

The M1/M2 have indeed some strange behaviour wrt OpenMP (I noticed quite some projects with similar issues). Thanks for trying so many options. As you figured out already, the compiler thinks omp is available but the library is missing. Brew installing omp should be sufficient (potentially together with brew install llvm) but on the new system macos appears more specific about the paths. I have updated the setup file accordingly.

But ... we recently got access to M1/M2 macs (Github Actions doesn't support them yet for automated compilation). So I was able to compile and upload to PyPI a precompiled wheel that should match your setup (Python 3.10 and macos-arm). This should now work by reinstalling (pip install --force-reinstall --no-cache dtaidistance) or by downloading the wheel from https://pypi.org/project/dtaidistance/#files (dtaidistance-2.3.9-cp310-cp310-macosx_12_0_arm64.whl).

(Unrelated, our unit tests run noticeably faster on the M1/M2 macs, so I hope you also experience this speed bump :-) )

tommedema commented 1 year ago

This sounds very promising, thanks @wannesm! Unfortunately when trying pip install --force-reinstall --no-cache dtaidistance I now get:

Cannot import OMP-based library (dtw_cc_omp)
Cannot import Numpy-based library (dtw_cc_numpy)

Not all libraries are available in your installation. 
You can rerun the compilation from source or pip install in verbose mode:
pip install -vvv --upgrade --force-reinstall --no-deps --no-binary dtaidistance dtaidistance
In case you need to use an older version of numpy, compile against your current installation:
pip install -vvv --upgrade --force-reinstall --no-deps --no-build-isolation --no-binary dtaidistance dtaidistance

Share the following information when submitting a bug report:
== Packages ==
- Cannot import OMP-based library (dtw_cc_omp)
- dlopen(/Users/playground/.pyenv/versions/3.10.6/lib/python3.10/site-packages/dtaidistance/dtw_cc_omp.cpython-310-darwin.so, 0x0002): Library not loaded: '/opt/local/lib/libomp/libomp.dylib'
  Referenced from: '/Users/playground/.pyenv/versions/3.10.6/lib/python3.10/site-packages/dtaidistance/dtw_cc_omp.cpython-310-darwin.so'
  Reason: tried: '/opt/local/lib/libomp/libomp.dylib' (no such file), '/usr/local/lib/libomp.dylib' (no such file), '/usr/lib/libomp.dylib' (no such file)
- Cannot import Numpy-based library (dtw_cc_numpy)
- dlopen(/Users/playground/.pyenv/versions/3.10.6/lib/python3.10/site-packages/dtaidistance/dtw_cc_numpy.cpython-310-darwin.so, 0x0002): Library not loaded: '/opt/local/lib/libomp/libomp.dylib'
  Referenced from: '/Users/playground/.pyenv/versions/3.10.6/lib/python3.10/site-packages/dtaidistance/dtw_cc_numpy.cpython-310-darwin.so'
  Reason: tried: '/opt/local/lib/libomp/libomp.dylib' (no such file), '/usr/local/lib/libomp.dylib' (no such file), '/usr/lib/libomp.dylib' (no such file)
- Numpy version: 1.23.3
- Matplotlib version: 3.6.0
- Scipy version: 1.9.1
== System information ==
namespace(name='cpython', cache_tag='cpython-310', version=sys.version_info(major=3, minor=10, micro=6, releaselevel='final', serial=0), hexversion=50988784, _multiarch='darwin')
== Compilation information ==
Compiler type: unix
--noopenmp: 0
--forceopenmp: 0
--noxpreprocessor: 0
--forcellvm: 0
--forcegnugcc: 0
Cython found (during compilation)
- Version: 0.29.32
- Locations: <module 'Cython' from '/Users/wannes/Projects/Research/2016-DTW/repo_dtw/venv-310/lib/python3.10/site-packages/Cython/__init__.py'>
Numpy found (during compilation):
- Version: 1.23.3
- Location: <module 'numpy' from '/Users/wannes/Projects/Research/2016-DTW/repo_dtw/venv-310/lib/python3.10/site-packages/numpy/__init__.py'>
Using LLVM settings (/usr/bin/clang)
Checking for OpenMP availability for /usr/bin/clang
/usr/bin/clang -dM -E -Xpreprocessor -fopenmp - # with stdout=-1, stderr=-1, input=, encoding=ascii

#define _OPENMP 201811
... found OpenMP
All extensions:
[<setuptools.extension.Extension('dtaidistance.dtw_cc') at 0x117f27640>, <setuptools.extension.Extension('dtaidistance.ed_cc') at 0x117f27b50>, <setuptools.extension.Extension('dtaidistance.dtw_cc_omp') at 0x117f34040>, <setuptools.extension.Extension('dtaidistance.dtw_cc_numpy') at 0x117f359c0>]

==
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
Cell In [2], line 4
      2 from dtaidistance import dtw
      3 if not dtw.try_import_c(verbose = False):
----> 4     raise Exception('Failed to import C version of dtaidistance')

Exception: Failed to import C version of dtaidistance

When I try to pull the new code and install from source, I still get:

% python3 setup.py build_ext --inplace

dtaidistance/dtw_cc.c:781:10: fatal error: 'omp.h' file not found
#include <omp.h>
         ^~~~~~~
1 error generated.
ERROR: The C extension could not be compiled
error: command '/usr/bin/clang' failed with exit code 1
running build_ext
Installed the plain Python version of the package.
If you need the C extension, try reinstalling.
dtaidistance-master 2 %

Perhaps relevant:

% brew list libomp 
/opt/homebrew/Cellar/libomp/14.0.6/include/ (3 files)
/opt/homebrew/Cellar/libomp/14.0.6/lib/libomp.dylib
/opt/homebrew/Cellar/libomp/14.0.6/lib/libomp.a

This is where homebrew has linked libomp:

% brew unlink libomp --dry-run | xargs ls -l
ls: Would: No such file or directory
ls: remove:: No such file or directory
lrwxr-xr-x  1 playground  admin  43 Sep 17 15:49 /opt/homebrew/include/omp-tools.h -> ../Cellar/libomp/14.0.6/include/omp-tools.h
lrwxr-xr-x  1 playground  admin  37 Sep 17 15:49 /opt/homebrew/include/omp.h -> ../Cellar/libomp/14.0.6/include/omp.h
lrwxr-xr-x  1 playground  admin  38 Sep 17 15:49 /opt/homebrew/include/ompt.h -> ../Cellar/libomp/14.0.6/include/ompt.h
lrwxr-xr-x  1 playground  admin  36 Sep 17 15:49 /opt/homebrew/lib/libomp.a -> ../Cellar/libomp/14.0.6/lib/libomp.a
lrwxr-xr-x  1 playground  admin  40 Sep 17 15:49 /opt/homebrew/lib/libomp.dylib -> ../Cellar/libomp/14.0.6/lib/libomp.dylib
tommedema commented 1 year ago

this started working when I did:

eval "$(/opt/homebrew/bin/brew shellenv)"

pip install -vvv --upgrade --force-reinstall --no-deps --no-build-isolation --no-binary dtaidistance git+https://github.com/wannesm/dtaidistance.git#egg=dtaidistance

I don't know why I have to run the eval statement since this is already on my path from ~/.zprofile

and somehow it was no longer enough to restart my notebook kernel -- I also had to completely stop the jupyter server, then restart it

note that I actually had to change it to git+https://github.com/wannesm/dtaidistance.git#egg=dtaidistance from dtaidistance or otherwise I would get this issue: TypeError: subsequence_search() got an unexpected keyword argument 'dists_options'

really appreciate your excellent help and advice

wannesm commented 1 year ago

The setup.py file in the master branch was corrected for M1/M2 and has picked up on your installation of libomp. Turns out the precompiled version was dynamically linking libomp and at a different location than the default homebrew location. Even though the static library was also available. Your detailed feedback made that clear and was thus very useful since I didn't realize before that macos had this default behavior.