Installation via pip on Python 3.9 and newer fails with fatal error: xtensor/xarray.hpp: No such file or directory

[fsimonjetz]

Hi there,

first of, this is an excellent library. I am utilizing it for a research project and had no issues installing and running it on Kaggle. Recently, however, Kaggle apparently upgraded to Python 3.10.10 and the installation no longer works. This is the error I'm getting when I run !pip install pyalign in a fresh notebook:

Collecting pyalign
  Using cached pyalign-0.4.4.tar.gz (50 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: numpy~=1.19 in /opt/conda/lib/python3.10/site-packages (from pyalign) (1.23.5)
Collecting pybind11~=2.6.2 (from pyalign)
  Using cached pybind11-2.6.2-py2.py3-none-any.whl (191 kB)
Requirement already satisfied: cached-property in /opt/conda/lib/python3.10/site-packages (from pyalign) (1.5.2)
Collecting py-cpuinfo==8.0.0 (from pyalign)
  Using cached py_cpuinfo-8.0.0-py3-none-any.whl
Collecting pymorton==1.0.5 (from pyalign)
  Using cached pymorton-1.0.5-py2.py3-none-any.whl (4.4 kB)
Collecting pyyaml==6.0 (from pyalign)
  Using cached PyYAML-6.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (682 kB)
Collecting cpufeature==0.2.0 (from pyalign)
  Using cached cpufeature-0.2.0-cp310-cp310-linux_x86_64.whl
Building wheels for collected packages: pyalign
  Building wheel for pyalign (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Building wheel for pyalign (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [47 lines of output]
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-310
      creating build/lib.linux-x86_64-cpython-310/pyalign
      copying pyalign/cpu.py -> build/lib.linux-x86_64-cpython-310/pyalign
      copying pyalign/__init__.py -> build/lib.linux-x86_64-cpython-310/pyalign
      copying pyalign/simple.py -> build/lib.linux-x86_64-cpython-310/pyalign
      copying pyalign/solve.py -> build/lib.linux-x86_64-cpython-310/pyalign
      copying pyalign/gaps.py -> build/lib.linux-x86_64-cpython-310/pyalign
      copying pyalign/utils.py -> build/lib.linux-x86_64-cpython-310/pyalign
      copying pyalign/_version.py -> build/lib.linux-x86_64-cpython-310/pyalign
      creating build/lib.linux-x86_64-cpython-310/pyalign/tests
      copying pyalign/tests/test_nw.py -> build/lib.linux-x86_64-cpython-310/pyalign/tests
      copying pyalign/tests/test_gotoh.py -> build/lib.linux-x86_64-cpython-310/pyalign/tests
      copying pyalign/tests/__init__.py -> build/lib.linux-x86_64-cpython-310/pyalign/tests
      copying pyalign/tests/test_sw.py -> build/lib.linux-x86_64-cpython-310/pyalign/tests
      copying pyalign/tests/test_batch.py -> build/lib.linux-x86_64-cpython-310/pyalign/tests
      copying pyalign/tests/test_wsb.py -> build/lib.linux-x86_64-cpython-310/pyalign/tests
      creating build/lib.linux-x86_64-cpython-310/pyalign/io
      copying pyalign/io/alignment.py -> build/lib.linux-x86_64-cpython-310/pyalign/io
      copying pyalign/io/__init__.py -> build/lib.linux-x86_64-cpython-310/pyalign/io
      copying pyalign/io/plot.py -> build/lib.linux-x86_64-cpython-310/pyalign/io
      creating build/lib.linux-x86_64-cpython-310/pyalign/problems
      copying pyalign/problems/function.py -> build/lib.linux-x86_64-cpython-310/pyalign/problems
      copying pyalign/problems/__init__.py -> build/lib.linux-x86_64-cpython-310/pyalign/problems
      copying pyalign/problems/factory.py -> build/lib.linux-x86_64-cpython-310/pyalign/problems
      copying pyalign/problems/instance.py -> build/lib.linux-x86_64-cpython-310/pyalign/problems
      creating build/lib.linux-x86_64-cpython-310/pyalign/algorithm
      copying pyalign/algorithm/__init__.py -> build/lib.linux-x86_64-cpython-310/pyalign/algorithm
      running build_ext
      building 'pyalign.algorithm.native.algorithm' extension
      creating build/temp.linux-x86_64-cpython-310
      creating build/temp.linux-x86_64-cpython-310/tmp
      creating build/temp.linux-x86_64-cpython-310/tmp/pip-install-vv6368xe
      creating build/temp.linux-x86_64-cpython-310/tmp/pip-install-vv6368xe/pyalign_70ad6a0542354aa5a94076826e635912
      creating build/temp.linux-x86_64-cpython-310/tmp/pip-install-vv6368xe/pyalign_70ad6a0542354aa5a94076826e635912/pyalign
      creating build/temp.linux-x86_64-cpython-310/tmp/pip-install-vv6368xe/pyalign_70ad6a0542354aa5a94076826e635912/pyalign/algorithm
      gcc -pthread -B /opt/conda/compiler_compat -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /opt/conda/include -fPIC -O2 -isystem /opt/conda/include -fPIC -I/tmp/pip-build-env-lzyua9kw/overlay/lib/python3.10/site-packages/numpy/core/include -I/tmp/pip-install-vv6368xe/pyalign_70ad6a0542354aa5a94076826e635912 -I/tmp/pip-build-env-lzyua9kw/overlay/lib/python3.10/site-packages/pybind11/include -I/opt/conda/include/python3.10 -c /tmp/pip-install-vv6368xe/pyalign_70ad6a0542354aa5a94076826e635912/pyalign/algorithm/module.cpp -o build/temp.linux-x86_64-cpython-310/tmp/pip-install-vv6368xe/pyalign_70ad6a0542354aa5a94076826e635912/pyalign/algorithm/module.o -fvisibility=hidden -g0 -std=c++17 -O3 -ftemplate-backtrace-limit=0 -march=native
      In file included from /tmp/pip-install-vv6368xe/pyalign_70ad6a0542354aa5a94076826e635912/pyalign/algorithm/pyalign.h:4,
                       from /tmp/pip-install-vv6368xe/pyalign_70ad6a0542354aa5a94076826e635912/pyalign/algorithm/module.cpp:3:
      /tmp/pip-install-vv6368xe/pyalign_70ad6a0542354aa5a94076826e635912/pyalign/algorithm/common.h:6:10: fatal error: xtensor/xarray.hpp: No such file or directory
          6 | #include <xtensor/xarray.hpp>
            |          ^~~~~~~~~~~~~~~~~~~~
      compilation terminated.
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pyalign
Failed to build pyalign
ERROR: Could not build wheels for pyalign, which is required to install pyproject.toml-based projects

I'm gettin the exact same error in a local environment running Python 3.9. Installation in 3.8.10 works fine though. Conda is not an option since we need to install the library in an existing project based on Poetry/PyPy.

Any help would be greatly appreciated. Thank you!

[MordorianGuy]

No way of installation works. I tried installing both via pip & locally but this ended up the same.

[firsc]

I fixed this for me by installing those libraries with conda:

mamba install -c conda-forge xtensor-python xsimd

After doing this, the pip install should work (if this is still not enough, there are more packages inside the environment.yml).

I believe there are also Ubuntu packages for most of these libraries (apt install xtensor-dev libxsimd-dev). The only library that seems to be missing via Ubuntu package is xtensor-python.

[pejrich]

@firsc Anyway to install that doesn't require conda/mamba?

[firsc]

@pejrich As I said, most missing packages are available as Ubuntu packages (apt install xtensor-dev libxsimd-dev). For the xtensor-python package, you can check here for instructions to build with cmake if you don't want to use conda.