The compiled dtaidistance C library is not available, cannot import Numpy-based library (dtw_cc_numpy)

[tommedema]

First, this library is amazing, thanks for all the hard work. 🙌

I'm on a Macbook Pro with macOS Monterey 12.5 and am having trouble enabling c for subsequence_alignment.

I've installed both with pip and from source, but the issue remains.

I've tried:

brew install libomp
pip install -vvv --upgrade --no-cache-dir --force-reinstall dtaidistance
pip install --global-option=--noopenmp -vvv --upgrade --no-cache-dir --force-reinstall dtaidistance
pip install -vvv --upgrade --force-reinstall --no-deps --no-binary dtaidistance dtaidistance

And installed from source too following the docs.

When I include dtw.try_import_c(verbose=True) this is the output:

Cannot import Numpy-based library (dtw_cc_numpy)

Not all libraries are available in your installation. 
You can rerun the compilation from source or pip install in verbose mode:
pip install -vvv --upgrade --force-reinstall --no-deps --no-binary dtaidistance dtaidistance
In case you need to use an older version of numpy, compile against your current installation:
pip install -vvv --upgrade --force-reinstall --no-deps --no-build-isolation --no-binary dtaidistance dtaidistance

Share the following information when submitting a bug report:
== Packages ==
- Cannot import Numpy-based library (dtw_cc_numpy)
- numpy.ndarray size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject
- Numpy version: 1.20.3
- Matplotlib version: 3.4.3
- Scipy version: 1.7.1
== System information ==
namespace(name='cpython', cache_tag='cpython-39', version=sys.version_info(major=3, minor=9, micro=7, releaselevel='final', serial=0), hexversion=50923504, _multiarch='darwin')
== Compilation information ==
Compiler type: unix
--noopenmp: 0
--forceopenmp: 0
--noxpreprocessor: 0
--forcellvm: 0
--forcegnugcc: 0
Cython found (during compilation)
- Version: 0.29.32
- Locations: <module 'Cython' from '/private/var/folders/lm/xhqw06kd341ck445l0rpz_cr0000gn/T/pip-build-env-q8jo9eja/overlay/lib/python3.9/site-packages/Cython/__init__.py'>
Numpy found (during compilation):
- Version: 1.23.1
- Location: <module 'numpy' from '/private/var/folders/lm/xhqw06kd341ck445l0rpz_cr0000gn/T/pip-build-env-q8jo9eja/overlay/lib/python3.9/site-packages/numpy/__init__.py'>
Using LLVM settings (clang)
Checking for OpenMP availability for clang
clang -dM -E -Xpreprocessor -fopenmp - # with stdout=-1, stderr=-1, input=, encoding=ascii

#define _OPENMP 201811
... found OpenMP
All extensions:
[<setuptools.extension.Extension('dtaidistance.dtw_cc') at 0x7f7f68152fd0>, <setuptools.extension.Extension('dtaidistance.ed_cc') at 0x7f7f681856d0>, <setuptools.extension.Extension('dtaidistance.dtw_cc_omp') at 0x7f7f68185b50>, <setuptools.extension.Extension('dtaidistance.dtw_cc_numpy') at 0x7f7f6819e910>]

==
---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
/var/folders/lm/xhqw06kd341ck445l0rpz_cr0000gn/T/ipykernel_39412/2171289896.py in <module>
      8 overlap = False
      9 
---> 10 sa = subsequence_alignment(query, series, use_c = True)
     11 
     12 kmatches = list(sa.kbest_matches(k = k, overlap = overlap))

~/opt/anaconda3/lib/python3.9/site-packages/dtaidistance/subsequence/dtw.py in subsequence_alignment(query, series, use_c)
     56     """
     57     sa = SubsequenceAlignment(query, series, use_c=use_c)
---> 58     sa.align()
     59     return sa
     60 

~/opt/anaconda3/lib/python3.9/site-packages/dtaidistance/subsequence/dtw.py in align(self)
    135                                                   psi_neg=False)
    136             else:
--> 137                 _, self.paths = dtw.warping_paths_fast(self.query, self.series, penalty=self.penalty, psi=psi,
    138                                                        compact=False, psi_neg=False)
    139         self._compute_matching()

~/opt/anaconda3/lib/python3.9/site-packages/dtaidistance/dtw.py in warping_paths_fast(s1, s2, window, max_dist, use_pruning, max_step, max_length_diff, penalty, psi, psi_neg, compact, use_ndim)
    479     r = len(s1)
    480     c = len(s2)
--> 481     _check_library(raise_exception=True)
    482     if window is None:
    483         window = 0

~/opt/anaconda3/lib/python3.9/site-packages/dtaidistance/dtw.py in _check_library(include_omp, raise_exception)
     92         logger.error(msg)
     93         if raise_exception:
---> 94             raise Exception(msg)
     95     if include_omp and (dtw_cc_omp is None or not dtw_cc_omp.is_openmp_supported()):
     96         msg = "The compiled dtaidistance C-OMP library "

Exception: The compiled dtaidistance C library is not available.
See the documentation for alternative installation options.

[wannesm]

Thanks for the detailed information. The installed numpy version is not binary compatible with the newest version (which is used to compile the software because pip uses isolated builds by default nowadays). Adding --no-build-isolation to the pip command should resolve this (or updating numpy). It then uses the version you have installed. Let me know if it doesn’t work.

[tommedema]

@wannesm hmm, so I already have the latest version of numpy (tried to reinstall it too and force upgrade with pip install numpy --upgrade).

I then tried --no-build-isolation too:

pip install -vvv --upgrade --force-reinstall --no-deps --no-build-isolation --no-binary dtaidistance dtaidistance

Here's the output:

https://gist.github.com/tommedema/abdfcc7f7fa3266ce809e25d5c581be4

When I run my notebook I still get:

Cannot import Numpy-based library (dtw_cc_numpy) [ full output ]

I also tried using pip3 instead of pip, but this made no difference.

Is it possible that my Jupyter notebook is picking up the dtaidistance I compiled myself rather than the one pip is trying to re-install? If so how would I uninstall the version I compiled myself?

[tommedema]

@wannesm actually, it worked! I had to restart my notebook's kernel. Thank you so much.