search

CannyLab / tsne-cuda

GPU Accelerated t-SNE for CUDA with Python bindings
BSD 3-Clause "New" or "Revised" License
1.81k stars 130 forks source link

faiss installed from the channel conda-forge breaks tsnecuda #91

Closed hkunzhe closed 3 years ago

hkunzhe commented 3 years ago

Since faiss-gpu doesn't provide compiled binaries with cuda10.1, 10.2, and 11.0, see this issue for details. Thus, I install faiss from the channel conda-forge by:

There are some already compiled builds for cuda 10.1, 10.2, and 11.0 in the conda-forge channel. To install a specific build, try conda search faiss= --info -c conda-forge and find the no-conflict one with your environment. Then you can install by conda install faiss==, e.g., conda install faiss=1.6.3=py37hddabd6f_3_cuda -c conda-forge.

However, when I run:

import tsnecuda
tsnecuda.test()

It shows an OSError: anaconda3/envs/test/lib/python3.7/site-packages/tsnecuda/libtsnecuda.so: undefined symbol: _ZTIN5faiss11IndexShardsE.

Steps to reproduce:

  1. conda create -n test python=3.7 and conda activate test.
  2. conda install tsnecuda cuda101 -c cannylab.
  3. conda install faiss=1.6.3=py37hddabd6f_3_cuda -c conda-forge.
  4. run tsnecuda.test().
hkunzhe commented 3 years ago

It's a similar issue with #65. I've tested faiss-gpu with cudatoolkit 10.0 and tsnecuda with cuda100 working fine.

DavidMChan commented 3 years ago

Yes - I think it remains outstanding to build our latest version of the code linking dynamically against the faiss-gpu libraries from conda-forge.

Our current build from source will build faiss-GPU in addition to tsnecuda, linking against the local version. If you have a second version of faiss resident in conda, I'm not sure exactly what happens - it likely tries to link against the conda-forge version, which is different from the static symbol compiled into the local version of the code.

The best solution seems to be to build tsnecuda from source, and alter the CMakeLists.txt to link against faiss-GPU as a dynamic library instead of linking statically against a locally compiled version. At the time we wrote this library, this wasn't an option, but it seems like it may be possible now.

hkunzhe commented 3 years ago

Ok, I got it! Thank you very much:)