Build ngtpy with pybind11

[VarIr]

How can I build ngtpy with pybind11 from source? Following the instruction from the readme, I am only able to build ngt with ctypes.

Background: I'd like to include ngtpy in one of my projects. On my machine I can simply pip install ngt, and I get ngtpy. However, the same gives me segmentation faults and others errors on Travis CI. So I try to install from source, but I only manage to get ngt (ctypes).

[masajiro]

Thank you for your interest in NGT!

Which os do you use, Linux or macOS? What version of NGT do you use? v1.7.9? What is the version of your python?

I am only able to build ngt with ctypes

Although there is the only way to install ngt (ctypes) and ngtpy, how do you know not to build ngtpy. Does it mean that you cannot import ngtpy in python? If so, what messages do you have? Did you uninstall ngtpy from PyPI before build and installation of ngtpy?

I am also concerned about your PyPI problem. NGTs of less than v1.7.8 from PyPI cause segmentation faults for specific cpus. Which version of NGT from PyPI do you use? What is your cpu?

[VarIr]

Thanks for the quick reply!

Which os do you use, Linux or macOS?

I develop under Linux, Fedora 29, kernel 5.1 on an Intel(R) Core(TM) i5-5300U CPU. Everything works fine there. On Travis, I use both Linux and macOS. See travis_cpu.txt for /proc/cpuinfo output on Linux. MacOS says machdep.cpu.brand_string: Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz. I'm not sure, whether Travis always uses the same CPU type, though...

When I pip install ngt on Linux, this installs ngt-1.7.9-cp37-cp37m-manylinux1_x86_64.whl, and pytest errors with segmentation fault during collection. So it seems, the error happens on import. On macOS pip installs ngt-1.5.1-cp37-cp37m-macosx_10_13_x86_64.whl, and gives the following error:

    import ngtpy
E   ImportError: dlopen(/Users/travis/miniconda/lib/python3.7/site-packages/ngtpy.cpython-37m-darwin.so, 2): Library not loaded: /usr/local/opt/gcc/lib/gcc/8/libgomp.1.dylib
E     Referenced from: /Users/travis/miniconda/lib/python3.7/site-packages/ngtpy.cpython-37m-darwin.so
E     Reason: image not found

What version of NGT do you use? v1.7.9?

On Linux pip seems to install v1.7.9, but on MacOS it installs the older version v.1.5.1. When I build from source, I fetch the latest release (v.1.7.9), but the internal version number is v.1.5.0. This seems to be hard-coded in setup.py.

What is the version of your python?

On Travis I have Python 3.7.3 for both Linux and MacOS, on my personal machine I have Python 3.7.4.

how do you know not to build ngtpy. Does it mean that you cannot import ngtpy in python?

Yes, you are correct. I jumped to that conclusion, because I could not import ngtpy in python. Error message on Travis, Linux, with ngtpy built from source:

    import ngtpy
E   ImportError: libngt.so.1: cannot open shared object file: No such file or directory

Did you uninstall ngtpy from PyPI before build and installation of ngtpy?

No, I did not explicitly uninstall ngtpy. It might have been cached on Travis. I've just tried now to uninstall first, then build from source. It seems, the error remains.

[masajiro]

Did you set LD_LIBRARY_PATH or use ldconfig to be able to find libngt? If not, please set LD_LIBRARY_PATH. By default, libngt must be found in /usr/local/lib or /usr/local/lib64, when you install NGT from source.

export LD_LIBRARY_PATH=/usr/local/lib

or

export LD_LIBRARY_PATH=/usr/local/lib64

[VarIr]

Great, it works fine now under Linux.

I'm still trying to figure out MacOS, though. If I may use some more of your time, could you please check the compile errors I get: Travis log (bottom)

Thanks a lot for your awesome tool and your help! Highly appreciated.

[masajiro]

Since I might find a kind of bug around definitions for AVX, I will fix it. To avoid it right away, you might want to swap this line with the line below.

cmake -DNGT_AVX_DISABLED=ON ..

[masajiro]

I'm still trying to figure out MacOS, though. If I may use some more of your time, could you please check the compile errors I get: Travis log (bottom)

Since I might find a kind of bug around definitions for AVX, I will fix it.

I released NGT v1.7.10 not to cause this problem.