yahoojapan / NGT

Nearest Neighbor Search with Neighborhood Graph and Tree for High-dimensional Data
Apache License 2.0
1.24k stars 114 forks source link

ctype base example only work in the library folder #28

Closed tungnt55 closed 5 years ago

tungnt55 commented 5 years ago

I am trying to run the example code for ctype in this guide below: https://github.com/yahoojapan/NGT/blob/master/python/README.md
I followed this instructions to install NGT for ctype.
https://github.com/yahoojapan/NGT/blob/master/README.md#build
However, since I don't have permission to write into /usr/local/lib/ folder, I had to run the installation like follow:

 $ tar zxvf NGT-1.7.6.tar.gz
 $ cd NGT-1.7.6
 $ mkdir build
 $ cd build 
 $ cmake -DCMAKE_INSTALL_PREFIX=<MY_NGT_DIR> ..
 $ make 
 $ make install

Then I set LD_LIBRARY_PATH in ~/.bashrc pointing to /lib/ folder and run source ~/.bashrc.
Finally, I run "pip install ngt". I tried to run the ctype example code but it gave me error like below:

AttributeError: python: undefined symbol: ngt_open_index

However, if I put the example code file in the /lib/ folder, it runs without problems.

masajiro commented 5 years ago

Although I followed your procedure to install NGT into a private directory, I could not find any problem on ubunutu 18.10 you mentioned. Why don't you check if LD_LIBRARY_PATH is adequately set with printenv? In addition, I recommend you to use ngtpy instead of NGT for ctypes because you get its better performance without installation of NGT libs.

tungnt55 commented 5 years ago

Although I followed your procedure to install NGT into a private directory, I could not find any problem on ubunutu 18.10 you mentioned. Why don't you check if LD_LIBRARY_PATH is adequately set with printenv? In addition, I recommend you to use ngtpy instead of NGT for ctypes because you get its better performance without installation of NGT libs.

Thank you very much for the reply.
Actually, I found out the problem now. In this file here, line 88:
https://github.com/yahoojapan/NGT/blob/master/python/ngt/base.py
The libngt.so library is loaded using ctypes.util.find_library() function. Unfortunately, this function does not lookup for .so libraries listed in LD_LIBRARY_PATH in python 3.5 (the one I'm using now). This function uses ldconfig to check for libraries listed in /etc/ld.so.cache, which I don't have permission to write to. Thus, libngt.so cannot be loaded and gave me error. In the later versions of python (3.6+), they fixed this problem so I think this issue can be closed now.
I'd like to use ngtpy as well, however, the project I'm working on is created by a colleague who used ngt.base in their scripts. I came accross this error while trying to reproduce the results in their work.

masajiro commented 5 years ago

Thank you for your helpful information. I did not know the bug in find_library().