Unable to run faiss on termux

[sheneeb]

After installing from source and running the following commands generates error On termux

python -c "import faiss; print(faiss.version)" Traceback (most recent call last): File "", line 1, in File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/faiss/init.py", line 16, in from .loader import File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/faiss/loader.py", line 98, in from .swigfaiss import File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/faiss/swigfaiss.py", line 10, in from . import _swigfaiss ImportError: dlopen failed: cannot locate symbol "PyByteArray_Type" referenced by "/data/data/com.termux/files/usr/lib/python3.11/site-packages/faiss/_swigfaiss.so"...

[kyamagu]

Could you share the install log with a verbose message; i.e., pip install -v faiss-cpu ?

[sheneeb]

$ pip install -v faiss-cpu Using pip 24.1 from /data/data/com.termux/files/usr/lib/python3.11/site-packages/pip (python 3.11) DEPRECATION: Loading egg at /data/data/com.termux/files/usr/lib/python3.11/site-packages/statsmodels-0.0.0-py3.11-linux-aarch64.egg is deprecated. pip 24.3 will enforce this behaviour change. A possible replacement is to use pip for package installation. Discussion can be found at https://github.com/pypa/pip/issues/12330 DEPRECATION: Loading egg at /data/data/com.termux/files/usr/lib/python3.11/site-packages/scikit_learn-1.5.0-py3.11-linux-aarch64.egg is deprecated. pip 24.3 will enforce this behaviour change. A possible replacement is to use pip for package installation. Discussion can be found at https://github.com/pypa/pip/issues/12330 DEPRECATION: Loading egg at /data/data/com.termux/files/usr/lib/python3.11/site-packages/threadpoolctl-3.5.0-py3.11.egg is deprecated. pip 24.3 will enforce this behaviour change. A possible replacement is to use pip for package installation. Discussion can be found at https://github.com/pypa/pip/issues/12330 DEPRECATION: Loading egg at /data/data/com.termux/files/usr/lib/python3.11/site-packages/joblib-1.4.2-py3.11.egg is deprecated. pip 24.3 will enforce this behaviour change. A possible replacement is to use pip for package installation. Discussion can be found at https://github.com/pypa/pip/issues/12330 Collecting faiss-cpu Downloading faiss_cpu-1.8.0.post1.tar.gz (63 kB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 63.7/63.7 kB 1.4 MB/s eta 0:00:00 Running command pip subprocess to install build dependencies Using pip 24.1 from /data/data/com.termux/files/usr/lib/python3.11/site-packages/pip (python 3.11) Collecting setuptools Obtaining dependency information for setuptools from https://files.pythonhosted.org/packages/55/b3/b3a3415d2debd837106ed417f8681d8af63037fed367fa1b85dbfef081f1/setuptools-70.1.0-py3-none-any.whl.metadata Using cached setuptools-70.1.0-py3-none-any.whl.metadata (6.0 kB) Collecting wheel Obtaining dependency information for wheel from https://files.pythonhosted.org/packages/7d/cd/d7460c9a869b16c3dd4e1e403cce337df165368c71d6af229a74699622ce/wheel-0.43.0-py3-none-any.whl.metadata Using cached wheel-0.43.0-py3-none-any.whl.metadata (2.2 kB) Collecting oldest-supported-numpy Obtaining dependency information for oldest-supported-numpy from https://files.pythonhosted.org/packages/dc/5c/e3c84cfdd488701aa074b22cf5bd227fb15d26e1d55a66d9088c39afa123/oldest_supported_numpy-2023.12.21-py3-none-any.whl.metadata Using cached oldest_supported_numpy-2023.12.21-py3-none-any.whl.metadata (9.8 kB) Link requires a different Python (3.11.9 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/3a/be/650f9c091ef71cb01d735775d554e068752d3ff63d7943b26316dc401749/numpy-1.21.2.zip (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11) Link requires a different Python (3.11.9 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/5f/d6/ad58ded26556eaeaa8c971e08b6466f17c4ac4d786cd3d800e26ce59cc01/numpy-1.21.3.zip (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11) Link requires a different Python (3.11.9 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/fb/48/b0708ebd7718a8933f0d3937513ef8ef2f4f04529f1f66ca86d873043921/numpy-1.21.4.zip (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11) Link requires a different Python (3.11.9 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/c2/a8/a924a09492bdfee8c2ec3094d0a13f2799800b4fdc9c890738aeeb12c72e/numpy-1.21.5.zip (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11) Link requires a different Python (3.11.9 not in: '>=3.7,<3.11'): https://files.pythonhosted.org/packages/45/b7/de7b8e67f2232c26af57c205aaad29fe17754f793404f59c8a730c7a191a/numpy-1.21.6.zip (from https://pypi.org/simple/numpy/) (requires-python:>=3.7,<3.11) Collecting numpy==1.23.2 (from oldest-supported-numpy) Using cached numpy-1.23.2-cp311-cp311-linux_aarch64.whl Using cached setuptools-70.1.0-py3-none-any.whl (882 kB) Using cached wheel-0.43.0-py3-none-any.whl (65 kB) Using cached oldest_supported_numpy-2023.12.21-py3-none-any.whl (4.9 kB) DEPRECATION: Loading egg at /data/data/com.termux/files/usr/lib/python3.11/site-packages/statsmodels-0.0.0-py3.11-linux-aarch64.egg is deprecated. pip 24.3 will enforce this behaviour change. A possible replacement is to use pip for package installation. Discussion can be found at https://github.com/pypa/pip/issues/12330 DEPRECATION: Loading egg at /data/data/com.termux/files/usr/lib/python3.11/site-packages/scikit_learn-1.5.0-py3.11-linux-aarch64.egg is deprecated. pip 24.3 will enforce this behaviour change. A possible replacement is to use pip for package installation. Discussion can be found at https://github.com/pypa/pip/issues/12330 DEPRECATION: Loading egg at /data/data/com.termux/files/usr/lib/python3.11/site-packages/threadpoolctl-3.5.0-py3.11.egg is deprecated. pip 24.3 will enforce this behaviour change. A possible replacement is to use pip for package installation. Discussion can be found at https://github.com/pypa/pip/issues/12330 DEPRECATION: Loading egg at /data/data/com.termux/files/usr/lib/python3.11/site-packages/joblib-1.4.2-py3.11.egg is deprecated. pip 24.3 will enforce this behaviour change. A possible replacement is to use pip for package installation. Discussion can be found at https://github.com/pypa/pip/issues/12330 Installing collected packages: wheel, setuptools, numpy, oldest-supported-numpy Creating /data/data/com.termux/files/usr/tmp/pip-build-env-gmn7yfoh/overlay/bin changing mode of /data/data/com.termux/files/usr/tmp/pip-build-env-gmn7yfoh/overlay/bin/wheel to 755 changing mode of /data/data/com.termux/files/usr/tmp/pip-build-env-gmn7yfoh/overlay/bin/f2py to 755 changing mode of /data/data/com.termux/files/usr/tmp/pip-build-env-gmn7yfoh/overlay/bin/f2py3 to 755 changing mode of /data/data/com.termux/files/usr/tmp/pip-build-env-gmn7yfoh/overlay/bin/f2py3.11 to 755 ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. seaborn 0.13.2 requires matplotlib!=3.6.1,>=3.4, which is not installed. mplfinance 0.12.10b0 requires matplotlib, which is not installed. Successfully installed numpy-1.23.2 oldest-supported-numpy-2023.12.21 setuptools-70.1.0 wheel-0.43.0 Installing build dependencies ... done Running command Getting requirements to build wheel running egg_info writing faiss_cpu.egg-info/PKG-INFO writing dependency_links to faiss_cpu.egg-info/dependency_links.txt writing requirements to faiss_cpu.egg-info/requires.txt writing top-level names to faiss_cpu.egg-info/top_level.txt reading manifest file 'faiss_cpu.egg-info/SOURCES.txt' adding license file 'LICENSE' writing manifest file 'faiss_cpu.egg-info/SOURCES.txt' Getting requirements to build wheel ... done Running command Preparing metadata (pyproject.toml) running dist_info creating /data/data/com.termux/files/usr/tmp/pip-modern-metadata-utrauc20/faiss_cpu.egg-info writing /data/data/com.termux/files/usr/tmp/pip-modern-metadata-utrauc20/faiss_cpu.egg-info/PKG-INFO writing dependency_links to /data/data/com.termux/files/usr/tmp/pip-modern-metadata-utrauc20/faiss_cpu.egg-info/dependency_links.txt writing requirements to /data/data/com.termux/files/usr/tmp/pip-modern-metadata-utrauc20/faiss_cpu.egg-info/requires.txt writing top-level names to /data/data/com.termux/files/usr/tmp/pip-modern-metadata-utrauc20/faiss_cpu.egg-info/top_level.txt writing manifest file '/data/data/com.termux/files/usr/tmp/pip-modern-metadata-utrauc20/faiss_cpu.egg-info/SOURCES.txt' reading manifest file '/data/data/com.termux/files/usr/tmp/pip-modern-metadata-utrauc20/faiss_cpu.egg-info/SOURCES.txt' adding license file 'LICENSE' writing manifest file '/data/data/com.termux/files/usr/tmp/pip-modern-metadata-utrauc20/faiss_cpu.egg-info/SOURCES.txt' creating '/data/data/com.termux/files/usr/tmp/pip-modern-metadata-utrauc20/faiss_cpu-1.8.0.post1.dist-info' Preparing metadata (pyproject.toml) ... done Requirement already satisfied: numpy<2.0,>=1.0 in /data/data/com.termux/files/usr/lib/python3.11/site-packages (from faiss-cpu) (1.26.4) Requirement already satisfied: packaging in /data/data/com.termux/files/usr/lib/python3.11/site-packages (from faiss-cpu) (23.2) Building wheels for collected packages: faiss-cpu Running command Building wheel for faiss-cpu (pyproject.toml) running bdist_wheel running build running build_py running build_ext building 'faiss._swigfaiss' extension swigging faiss/faiss/python/swigfaiss.i to faiss/faiss/python/swigfaiss_wrap.cpp swig -python -c++ -Doverride= -doxygen -Ifaiss -I/data/data/com.termux/files/usr/tmp/pip-build-env-gmn7yfoh/overlay/lib/python3.11/site-packages/numpy/core/include -Ifaiss -I/usr/local/include -DSWIGWORDSIZE64 -o faiss/faiss/python/swigfaiss_wrap.cpp faiss/faiss/python/swigfaiss.i faiss/faiss/python/swigfaiss.i:276: Error: Unable to find 'faiss/impl/platform_macros.h' faiss/faiss/python/swigfaiss.i:280: Error: Unable to find 'faiss/utils/ordered_key_value.h' faiss/faiss/python/swigfaiss.i:281: Error: Unable to find 'faiss/utils/Heap.h' faiss/faiss/python/swigfaiss.i:286: Error: Unable to find 'faiss/utils/AlignedTable.h' faiss/faiss/python/swigfaiss.i:287: Error: Unable to find 'faiss/utils/partitioning.h' faiss/faiss/python/swigfaiss.i:288: Error: Unable to find 'faiss/utils/hamming.h' faiss/faiss/python/swigfaiss.i:289: Error: Unable to find 'faiss/utils/hamming_distance/common.h' faiss/faiss/python/swigfaiss.i:394: Error: Unable to find 'faiss/utils/utils.h' faiss/faiss/python/swigfaiss.i:398: Error: Unable to find 'faiss/utils/distances.h' faiss/faiss/python/swigfaiss.i:399: Error: Unable to find 'faiss/utils/random.h' faiss/faiss/python/swigfaiss.i:400: Error: Unable to find 'faiss/utils/sorting.h' faiss/faiss/python/swigfaiss.i:402: Error: Unable to find 'faiss/MetricType.h' faiss/faiss/python/swigfaiss.i:407: Error: Unable to find 'faiss/Index.h' faiss/faiss/python/swigfaiss.i:409: Error: Unable to find 'faiss/impl/DistanceComputer.h' faiss/faiss/python/swigfaiss.i:412: Error: Unable to find 'faiss/IndexFlatCodes.h' faiss/faiss/python/swigfaiss.i:413: Error: Unable to find 'faiss/IndexFlat.h' faiss/faiss/python/swigfaiss.i:414: Error: Unable to find 'faiss/Clustering.h' faiss/faiss/python/swigfaiss.i:416: Error: Unable to find 'faiss/utils/extra_distances.h' faiss/faiss/python/swigfaiss.i:420: Error: Unable to find 'faiss/impl/Quantizer.h' faiss/faiss/python/swigfaiss.i:421: Error: Unable to find 'faiss/impl/ProductQuantizer.h' faiss/faiss/python/swigfaiss.i:422: Error: Unable to find 'faiss/impl/AdditiveQuantizer.h' faiss/faiss/python/swigfaiss.i:423: Error: Unable to find 'faiss/impl/residual_quantizer_encode_steps.h' faiss/faiss/python/swigfaiss.i:424: Error: Unable to find 'faiss/impl/ResidualQuantizer.h' faiss/faiss/python/swigfaiss.i:425: Error: Unable to find 'faiss/impl/LocalSearchQuantizer.h' faiss/faiss/python/swigfaiss.i:426: Error: Unable to find 'faiss/impl/ProductAdditiveQuantizer.h' faiss/faiss/python/swigfaiss.i:427: Error: Unable to find 'faiss/impl/CodePacker.h' faiss/faiss/python/swigfaiss.i:429: Error: Unable to find 'faiss/VectorTransform.h' faiss/faiss/python/swigfaiss.i:430: Error: Unable to find 'faiss/IndexPreTransform.h' faiss/faiss/python/swigfaiss.i:431: Error: Unable to find 'faiss/IndexRefine.h' faiss/faiss/python/swigfaiss.i:432: Error: Unable to find 'faiss/IndexLSH.h' faiss/faiss/python/swigfaiss.i:433: Error: Unable to find 'faiss/impl/PolysemousTraining.h' faiss/faiss/python/swigfaiss.i:434: Error: Unable to find 'faiss/IndexPQ.h' faiss/faiss/python/swigfaiss.i:435: Error: Unable to find 'faiss/IndexAdditiveQuantizer.h' faiss/faiss/python/swigfaiss.i:436: Error: Unable to find 'faiss/impl/io.h' faiss/faiss/python/swigfaiss.i:438: Error: Unable to find 'faiss/invlists/InvertedLists.h' faiss/faiss/python/swigfaiss.i:439: Error: Unable to find 'faiss/invlists/InvertedListsIOHook.h' faiss/faiss/python/swigfaiss.i:441: Error: Unable to find 'faiss/invlists/BlockInvertedLists.h' faiss/faiss/python/swigfaiss.i:442: Error: Unable to find 'faiss/invlists/DirectMap.h' faiss/faiss/python/swigfaiss.i:443: Error: Unable to find 'faiss/IndexIVF.h' faiss/faiss/python/swigfaiss.i:448: Error: Unable to find 'faiss/IVFlib.h' faiss/faiss/python/swigfaiss.i:449: Error: Unable to find 'faiss/impl/ScalarQuantizer.h' faiss/faiss/python/swigfaiss.i:450: Error: Unable to find 'faiss/IndexScalarQuantizer.h' faiss/faiss/python/swigfaiss.i:451: Error: Unable to find 'faiss/IndexIVFSpectralHash.h' faiss/faiss/python/swigfaiss.i:452: Error: Unable to find 'faiss/IndexIVFAdditiveQuantizer.h' faiss/faiss/python/swigfaiss.i:453: Error: Unable to find 'faiss/impl/HNSW.h' faiss/faiss/python/swigfaiss.i:454: Error: Unable to find 'faiss/IndexHNSW.h' faiss/faiss/python/swigfaiss.i:456: Error: Unable to find 'faiss/impl/kmeans1d.h' faiss/faiss/python/swigfaiss.i:459: Error: Unable to find 'faiss/impl/NNDescent.h' faiss/faiss/python/swigfaiss.i:460: Error: Unable to find 'faiss/IndexNNDescent.h' faiss/faiss/python/swigfaiss.i:462: Error: Unable to find 'faiss/IndexIVFFlat.h' faiss/faiss/python/swigfaiss.i:466: Error: Unable to find 'faiss/impl/NSG.h' faiss/faiss/python/swigfaiss.i:477: Error: Unable to find 'faiss/IndexNSG.h' faiss/faiss/python/swigfaiss.i:482: Error: Unable to find 'faiss/invlists/OnDiskInvertedLists.h' faiss/faiss/python/swigfaiss.i:485: Error: Unable to find 'faiss/impl/lattice_Zn.h' faiss/faiss/python/swigfaiss.i:486: Error: Unable to find 'faiss/IndexLattice.h' faiss/faiss/python/swigfaiss.i:489: Error: Unable to find 'faiss/IndexIVFPQ.h' faiss/faiss/python/swigfaiss.i:490: Error: Unable to find 'faiss/IndexIVFPQR.h' faiss/faiss/python/swigfaiss.i:491: Error: Unable to find 'faiss/Index2Layer.h' faiss/faiss/python/swigfaiss.i:493: Error: Unable to find 'faiss/IndexFastScan.h' faiss/faiss/python/swigfaiss.i:494: Error: Unable to find 'faiss/IndexAdditiveQuantizerFastScan.h' faiss/faiss/python/swigfaiss.i:495: Error: Unable to find 'faiss/IndexPQFastScan.h' faiss/faiss/python/swigfaiss.i:500: Error: Unable to find 'faiss/impl/simd_result_handlers.h' faiss/faiss/python/swigfaiss.i:501: Error: Unable to find 'faiss/IndexIVFFastScan.h' faiss/faiss/python/swigfaiss.i:502: Error: Unable to find 'faiss/IndexIVFAdditiveQuantizerFastScan.h' faiss/faiss/python/swigfaiss.i:503: Error: Unable to find 'faiss/IndexIVFIndependentQuantizer.h' faiss/faiss/python/swigfaiss.i:505: Error: Unable to find 'faiss/IndexIVFPQFastScan.h' faiss/faiss/python/swigfaiss.i:506: Error: Unable to find 'faiss/utils/quantize_lut.h' faiss/faiss/python/swigfaiss.i:508: Error: Unable to find 'faiss/IndexBinary.h' faiss/faiss/python/swigfaiss.i:509: Error: Unable to find 'faiss/IndexBinaryFlat.h' faiss/faiss/python/swigfaiss.i:510: Error: Unable to find 'faiss/IndexBinaryIVF.h' faiss/faiss/python/swigfaiss.i:511: Error: Unable to find 'faiss/IndexBinaryFromFloat.h' faiss/faiss/python/swigfaiss.i:512: Error: Unable to find 'faiss/IndexBinaryHNSW.h' faiss/faiss/python/swigfaiss.i:513: Error: Unable to find 'faiss/IndexBinaryHash.h' faiss/faiss/python/swigfaiss.i:515: Error: Unable to find 'faiss/impl/ThreadedIndex.h' faiss/faiss/python/swigfaiss.i:519: Error: Unable to find 'faiss/IndexShards.h' faiss/faiss/python/swigfaiss.i:522: Error: Unable to find 'faiss/IndexShardsIVF.h' faiss/faiss/python/swigfaiss.i:524: Error: Unable to find 'faiss/IndexReplicas.h' faiss/faiss/python/swigfaiss.i:528: Error: Unable to find 'faiss/MetaIndexes.h' faiss/faiss/python/swigfaiss.i:530: Error: Unable to find 'faiss/IndexRowwiseMinMax.h' faiss/faiss/python/swigfaiss.i:540: Error: Unable to find 'faiss/impl/AuxIndexStructures.h' faiss/faiss/python/swigfaiss.i:541: Error: Unable to find 'faiss/impl/IDSelector.h' faiss/faiss/python/swigfaiss.i:543: Error: Unable to find 'faiss/IndexIDMap.h' faiss/faiss/python/swigfaiss.i:550: Error: Unable to find 'faiss/utils/approx_topk/mode.h' faiss/faiss/python/swigfaiss.i:792: Error: Unable to find 'faiss/index_io.h' faiss/faiss/python/swigfaiss.i:793: Error: Unable to find 'faiss/clone_index.h' faiss/faiss/python/swigfaiss.i:797: Error: Unable to find 'faiss/AutoTune.h' faiss/faiss/python/swigfaiss.i:798: Error: Unable to find 'faiss/index_factory.h' faiss/faiss/python/swigfaiss.i:799: Error: Unable to find 'faiss/MatrixStats.h' error: command '/data/data/com.termux/files/usr/bin/swig' failed with exit code 1 error: subprocess-exited-with-error

× Building wheel for faiss-cpu (pyproject.toml) did not run successfully. │ exit code: 1 ╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip. full command: /data/data/com.termux/files/usr/bin/python3.11 /data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /data/data/com.termux/files/usr/tmp/tmpdcad0atr cwd: /data/data/com.termux/files/usr/tmp/pip-install-_2rave66/faiss-cpu_2b4e1e90537e49cabbceb6ba9681cf73 Building wheel for faiss-cpu (pyproject.toml) ... error ERROR: Failed building wheel for faiss-cpu Failed to build faiss-cpu ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (faiss-cpu)

[sheneeb]

~ $ swig --version Unable to find option or file '--version', Use 'swig -help' for more information. ~ $ swig -version

SWIG Version 4.2.1

Compiled with aarch64-linux-android-clang++ [aarch64-unknown-linux-android]

Configured options: +pcre

Please see https://www.swig.org for reporting bugs and further information ~ $

[kyamagu]

Ok, have you installed faiss before building the source package?

https://github.com/kyamagu/faiss-wheels?tab=readme-ov-file#building-source-package

[sheneeb]

~ $ python Python 3.11.9 (main, Jun 10 2024, 00:57:33) [Clang 17.0.2 (https://android.googlesource.com/toolchain/llvm-project d9f89f4d1 on linux Type "help", "copyright", "credits" or "license" for more information.

import faiss Traceback (most recent call last): File "", line 1, in File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/faiss/init.py", line 16, in from .loader import File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/faiss/loader.py", line 98, in from .swigfaiss import File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/faiss/swigfaiss.py", line 10, in from . import _swigfaiss ImportError: dlopen failed: cannot locate symbol "PyByteArray_Type" referenced by "/data/data/com.termux/files/usr/lib/python3.11/site-packages/faiss/_swigfaiss.so"...

Installed from source but getting following error

[kyamagu]

The swig error is indicating the system does not have faiss installed. Build and install faiss first to build the source package. https://github.com/facebookresearch/faiss/blob/main/INSTALL.md#building-from-source