ImportError: cannot import name test_python_binding

[newling]

Both with a clean install from source, and using the package from the repo:

import mlpack
ImportError: cannot import name test_python_binding

..........:~$ uname -a
Linux idbean 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2 (2017-04-30) x86_64 GNU/Linux

[rcurtin]

Hi James,

Thanks for reporting the issue. The Python bindings are still being debugged so I am really happy to get feedback and work things like this out. Where did the Python bindings install, and where did libmlpack.so install?

My first guess is, try setting LD_LIBRARY_PATH to where libmlpack.so is:

export LD_LIBRARY_PATH=/path/to/libdir/

and try again. I am not sure if that will fix it, but let's see if it does. :)

[newling]

Hi Ryan,

export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu

doesn't seem to resolve the issue,

ImportError: cannot import name test_python_binding

but the issue might be elsewhere, when I add openblas lib path,

export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/:/home/james/openblas/lib/

I get a different error,

ImportError: /usr/lib/libblas.so.3: undefined symbol: gotoblas

Actually after adding the openblas path, numpy won't import either,

import numpy
.
.
.
ImportError: /usr/lib/libblas.so.3: undefined symbol: gotoblas

I'm not sure if this is related to the initial issue.

[rcurtin]

Hm, ok, thanks for the input. Can you tell me the steps you did to configure, build, and install mlpack? I will try and reproduce the issue. Thanks!

[newling]

I can't remember exactly what I did... but my first attempt was building from source from git, the usual

mkdir build 
cd build
cmake ..
make

with a few iterations of cmake to get all dependencies (armadillo, boost etc). But it seemed to work in the end. Then

sudo make install

I can't remember what the problem was after that, but I then removed everything from the install and used synaptic package manager, after which I got

ImportError: cannot import name test_python_binding

Once again this could be specific to me and my setup, I'll look into again more deeply at some point too...

Thanks!

[rcurtin]

Thanks for the info. I think what has happened here is that when you uninstalled the mlpack you built from source, not all files were removed from the Python directories. My guess is that /usr/local/lib/python2.7/dist-packages/mlpack/__init__.py still exists but nothing else is in that directory. (I'm guessing your Python version and I can't remember if it will install into dist-packages/ or site-packages/ so the exact path there may not be right...)

The version you installed from Synaptic is mlpack version 2, which doesn't have Python bindings. For the bindings you would need to install from source until stable mlpack 3.0.0 is released. If you did have problems doing that I can try and help you debug that also, just let me know.

[newling]

@rcurtin Thanks for this suggestion, I'll check this possibility out when I get a chance.

[newling]

So this time, I

1) removed everything 2) make install to a directory specified with CMAKE_INSTALL_PREFIX.

The only .so shared library file in the install dir is ./lib/libmlpack.so.2.0, I was expecting to see some kinda python shared library installed.

I see there are a bunch of .so files in ./build/src/mlpack/bindings/python/mlpack/ and cd-ing into that dir I can import knn without error but it looks like an empty libraty : help(knn) gives no info.

Hmm. I'm probably doing something wrong.

[rcurtin]

That all sounds correct to me. When building, all of the Python .so files end up exactly where you said---build/src/mlpack/bindings/python/mlpack/, and when you install them they all get moved to your Python installation directory. If you want to test the bindings without installing, you can just do (from the build directory):

export LD_LIBRARY_PATH=/path/to/build/lib/
export PYTHONPATH=/path/to/build/src/mlpack/bindings/python/mlpack/
python

At least, that's what I do. I looked into ways to automatically set the Python path when running Python from the build directory (so that one wouldn't have to manually set $PYTHONPATH like above), but I did not find a way to do it.

Once you've run Python like that, do you still have problems with import mlpack?

[newling]

when you install them they all get moved to your Python installation directory

I don't think so, unless there is a separate CMake directory I need to set for Python installs,

.../mlpack/install$ find . -name "*\.so*" 
./lib/libmlpack.so
./lib/libmlpack.so.2
./lib/libmlpack.so.2.0

From the build directory I'm not having any more luck:

$ ls /home/james/clustering/mlpack/mlpack/build/lib/
libmlpack.so  libmlpack.so.2  libmlpack.so.2.0  pkgconfig

$ ls /home/james/clustering/mlpack/mlpack/build/src/mlpack/bindings/python/mlpack/
adaboost.cpp    cli.pxd             fastmks.pyx          hmm_loglik.pyx      kfn.cpp     linear_regression.cpp        mean_shift.so   perceptron.pyx           softmax_regression.cpp
adaboost.pyx    cli_util.hpp        fastmks.so           hmm_loglik.so       kfn.pyx     linear_regression.pyx        move.hpp        perceptron.so            softmax_regression.pyx
adaboost.so     decision_stump.cpp  gmm_generate.cpp     hmm_train.cpp       kfn.so      linear_regression.so         nbc.cpp         preprocess_binarize.cpp  softmax_regression.so
approx_kfn.cpp  decision_stump.pyx  gmm_generate.pyx     hmm_train.pyx       kmeans.cpp  local_coordinate_coding.cpp  nbc.pyx         preprocess_binarize.pyx  sparse_coding.cpp
approx_kfn.pyx  decision_stump.so   gmm_generate.so      hmm_train.so        kmeans.pyx  local_coordinate_coding.pyx  nbc.so          preprocess_binarize.so   sparse_coding.pyx
approx_kfn.so   decision_tree.cpp   gmm_probability.cpp  hmm_viterbi.cpp     kmeans.so   local_coordinate_coding.so   nca.cpp         preprocess_describe.cpp  sparse_coding.so
arma_numpy.cpp  decision_tree.pyx   gmm_probability.pyx  hmm_viterbi.pyx     knn.cpp     logistic_regression.cpp      nca.pyx         preprocess_describe.pyx  test_python_binding.cpp
arma_numpy.pxd  decision_tree.so    gmm_probability.so   hmm_viterbi.so      knn.pyx     logistic_regression.pyx      nca.so          preprocess_describe.so   test_python_binding.pyx
arma_numpy.pyx  det.cpp             gmm_train.cpp        hoeffding_tree.cpp  knn.so      logistic_regression.so       nmf.cpp         preprocess_split.cpp     test_python_binding.so
arma_numpy.so   det.pyx             gmm_train.pyx        hoeffding_tree.pyx  krann.cpp   lsh.cpp                      nmf.pyx         preprocess_split.pyx
arma.pxd        det.so              gmm_train.so         hoeffding_tree.so   krann.pyx   lsh.pyx                      nmf.so          preprocess_split.so
arma_util.hpp   emst.cpp            hmm_generate.cpp     __init__.py         krann.so    lsh.so                       pca.cpp         radical.cpp
cf.cpp          emst.pyx            hmm_generate.pyx     kernel_pca.cpp      lars.cpp    matrix_utils.py              pca.pyx         radical.pyx
cf.pyx          emst.so             hmm_generate.so      kernel_pca.pyx      lars.pyx    mean_shift.cpp               pca.so          radical.so
cf.so           fastmks.cpp         hmm_loglik.cpp       kernel_pca.so       lars.so     mean_shift.pyx               perceptron.cpp  setup.cfg

$ export LD_LIBRARY_PATH=/home/james/clustering/mlpack/mlpack/build/lib/

$ export PYTHONPATH=/home/james/clustering/mlpack/mlpack/build/src/mlpack/bindings/python/mlpack/

$ python
Python 2.7.12 (default, Nov 19 2016, 06:48:10) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import mlpack
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named mlpack
>>> 

[rcurtin]

Ah, sorry for the slow response. I was hoping to respond with a notification that I'd fixed everything but the fix is not quite there yet. I mistyped the PYTHONPATH, no need for the trailing mlpack/:

export PYTHONPATH=/home/james/clustering/mlpack/mlpack/build/src/mlpack/bindings/python/

and that should work.

[rcurtin]

Ok, the issue should be fixed now, if you want to try the not-yet-merged bindings branch:

$ git remote add rcurtin https://github.com/rcurtin/mlpack
$ git fetch rcurtin
$ git checkout rcurtin/bindings

Then the problem you were having should be fixed. You can install to some custom directory, then if you have $LD_LIBRARY_PATH set to the directory that includes libmlpack.so (or libmlpack.so is already found by the system linker), and the set of directories in $PYTHONPATH includes the directory that holds the mlpack egg that got installed, then import mlpack should work just fine.

I'm sorry this whole debugging process took so long---it turns out there was a lot to dig to the bottom of and a lot of funky behavior by setuptools that I had to learn about...

[newling]

It works :) Thanks a lot.

[rcurtin]

Also fixed with #1125.

[ajkl]

I see this was resolved, but I am facing the same issue right now. I have set the right LD_LIBRARY_PATH and PYTHONPATH. I also tried copying the package to the site-packages as suggested here but cant get rid of the error.

python
Python 2.7.13 |Anaconda custom (64-bit)| (default, Dec 20 2016, 23:09:15)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import mlpack
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ajkale/mlpack/build/src/mlpack/bindings/python/mlpack/__init__.py", line 6, in <module>
    from .test_python_binding import test_python_binding
ImportError: cannot import name test_python_binding

[rcurtin]

Hmm, I tried reproducing this by running an Ubuntu 16.04 docker container, installing Anaconda, then installing all the mlpack dependencies and compiling. Then from the build directory I did the following:

root@0ce2517519a1:~/mlpack/build# export LD_LIBRARY_PATH=/root/mlpack/build/lib/
root@0ce2517519a1:~/mlpack/build# export PYTHONPATH=/root/mlpack/build/src/mlpack/bindings/python/
root@0ce2517519a1:~/mlpack/build# python
Python 2.7.14 |Anaconda, Inc.| (default, Oct 16 2017, 17:29:19) 
[GCC 7.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from mlpack import pca
>>> import mlpack
>>>

So it looks like it worked fine for me. My first thought in your case is, did CMake configure with the correct Python version? If you can paste the CMake output from the original configuration step I think we can see if it's the correct Anaconda-provided Python version.

[ajkl]

I did check that before posting. CMake did pick the right anaconda python

cmake -D BOOST_LIBRARYDIR=/usr/lib/x86_64-linux-gnu -D BUILD_PYTHON_BINDINGS=ON ../
-- cotire 1.7.10 loaded.
-- Armadillo libraries: /usr/lib/libarmadillo.so
-- Boost version: 1.55.0
-- Found the following Boost libraries:
--   program_options
--   unit_test_framework
--   serialization
-- gitversion.hpp is already up to date.
-- Found Python: /usr/local/anaconda2/bin/python
CMake Warning (dev) at src/mlpack/methods/neighbor_search/CMakeLists.txt:39 (get_property):
  Policy CMP0026 is not set: Disallow use of the LOCATION target property.
  Run "cmake --help-policy CMP0026" for policy details.  Use the cmake_policy
  command to set the policy and suppress this warning.

  The LOCATION property should not be read from target "mlpack_knn".  Use the
  target name directly with add_custom_command, or use the generator
  expression $<TARGET_FILE>, as appropriate.

This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at src/mlpack/methods/neighbor_search/CMakeLists.txt:47 (get_property):
  Policy CMP0026 is not set: Disallow use of the LOCATION target property.
  Run "cmake --help-policy CMP0026" for policy details.  Use the cmake_policy
  command to set the policy and suppress this warning.

  The LOCATION property should not be read from target "mlpack_kfn".  Use the
  target name directly with add_custom_command, or use the generator
  expression $<TARGET_FILE>, as appropriate.

This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning (dev) at src/mlpack/methods/rann/CMakeLists.txt:43 (get_property):
  Policy CMP0026 is not set: Disallow use of the LOCATION target property.
  Run "cmake --help-policy CMP0026" for policy details.  Use the cmake_policy
  command to set the policy and suppress this warning.

  The LOCATION property should not be read from target "mlpack_krann".  Use
  the target name directly with add_custom_command, or use the generator
  expression $<TARGET_FILE>, as appropriate.

This warning is for project developers.  Use -Wno-dev to suppress it.

CMake Warning at CMakeLists.txt:469 (message):
  txt2man not found; man pages will not be generated.

-- Configuring done
-- Generating done
-- Build files have been written to: /home/ajkale/mlpack/build

[rcurtin]

Are you sure you have LD_LIBRARY_PATH set right? What happens if you run ldd /home/ajkale/mlpack/build/src/mlpack/bindings/python/mlpack/test_python_binding.so?

[ajkl]

Yes the LD_LIBRARY_PATH seems to be ok. This is what I get ldd /home/ajkale/mlpack/build/src/mlpack/bindings/python/mlpack/test_python_binding.so linux-vdso.so.1 (0x00007fff01319000) libmlpack.so.2 => /usr/local/lib/libmlpack.so.2 (0x00007fa5bf51f000) libboost_serialization.so.1.55.0 => /usr/lib/x86_64-linux-gnu/libboost_serialization.so.1.55.0 (0x00007fa5bf29d000) libpython2.7.so.1.0 => /usr/local/anaconda2/lib/libpython2.7.so.1.0 (0x00007fa5beea0000) libstdc++.so.6 => /usr/local/anaconda2/lib/libstdc++.so.6 (0x00007fa5beae4000) libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fa5be7e3000) libgomp.so.1 => /usr/local/anaconda2/lib/libgomp.so.1 (0x00007fa5be5bf000) libgcc_s.so.1 => /usr/local/anaconda2/lib/libgcc_s.so.1 (0x00007fa5be3a8000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fa5be18b000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa5bdddf000) librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fa5bdbd7000) libarmadillo.so.4 => /usr/lib/libarmadillo.so.4 (0x00007fa5bd9d0000) libboost_program_options.so.1.55.0 => /usr/lib/x86_64-linux-gnu/libboost_program_options.so.1.55.0 (0x00007fa5bd757000) libboost_unit_test_framework.so.1.55.0 => /usr/lib/x86_64-linux-gnu/libboost_unit_test_framework.so.1.55.0 (0x00007fa5bd4a9000) /lib64/ld-linux-x86-64.so.2 (0x00005596be49d000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fa5bd2a4000) libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fa5bd0a1000) libblas.so.3 => /usr/lib/libblas.so.3 (0x00007fa5bce62000) liblapack.so.3 => /usr/lib/liblapack.so.3 (0x00007fa5bc666000) libarpack.so.2 => /usr/lib/libarpack.so.2 (0x00007fa5bc418000) libatlas.so.3 => /usr/lib/libatlas.so.3 (0x00007fa5bbe79000) libgfortran.so.3 => /usr/local/anaconda2/lib/libgfortran.so.3 (0x00007fa5bbb70000) libquadmath.so.0 => /usr/local/anaconda2/lib/libquadmath.so.0 (0x00007fa5bb930000)

On Wed, Jan 24, 2018 at 12:01 PM Ryan Curtin notifications@github.com wrote:

Are you sure you have LD_LIBRARY_PATH set right? What happens if you run ldd /home/ajkale/mlpack/build/src/mlpack/bindings/python/mlpack/test_python_binding.so ?

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/mlpack/mlpack/issues/1122#issuecomment-360255755, or mute the thread https://github.com/notifications/unsubscribe-auth/AEsHv326mQVNBAMJ_Hphs1s1XC5Z1Pk-ks5tN4wBgaJpZM4PaIle .

[rcurtin]

Just to double-check, is /usr/local/lib/libmlpack.so.2 the version that you just built and installed, or is that perhaps another version of mlpack installed previously?

Debugging these types of issues in Python I have found to be difficult because Python doesn't give much for error output. Another thing to check is to make sure test_python_binding.so is in the mlpack/ directory that is on the PYTHONPATH. Let me know what you find and I'll try to keep thinking about other possible causes.

[ajkl]

Yes, I re-installed it to make sure the libmlpack.so* are created after the fresh install.

Answer to other questions -

ajkale@ai-ml-1:~/mlpack/build$ export LD_LIBRARY_PATH=/usr/local/lib/
ajkale@ai-ml-1:~/mlpack/build$ ls $LD_LIBRARY_PATH|grep mlpack
libmlpack.so
libmlpack.so.2
libmlpack.so.2.0
ajkale@ai-ml-1:~/mlpack/build$ mlpack_knn --help
k-Nearest-Neighbors Search

  This program will calculate the k-nearest-neighbors of a set of points using
  kd-trees or cover trees (cover tree support is experimental and may be slow).
  You may specify a separate set of reference points and query points, or just a
  reference set which will be used as both the reference and query set.
...

ajkale@ai-ml-1:~/mlpack/build$ export PYTHONPATH=/home/ajkale/mlpack/build/src/mlpack/bindings/python/

ajkale@ai-ml-1:~/mlpack/build$ python
Python 2.7.13 |Anaconda custom (64-bit)| (default, Dec 20 2016, 23:09:15)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://anaconda.org
>>> import mlpack
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ajkale/mlpack/build/src/mlpack/bindings/python/mlpack/__init__.py", line 6, in <module>
    from .test_python_binding import test_python_binding
ImportError: cannot import name test_python_binding
>>>

ajkale@ai-ml-1:~/mlpack/build$ ls $PYTHONPATH/mlpack | grep test_python_binding
test_python_binding.cpp
test_python_binding.pyx
test_python_binding.so

[rcurtin]

I thought about it for a while, and I'm not sure what exactly is going on. I'd like to reproduce this; can you tell me how you built this system so I can do the same?

[rcurtin]

I'm actually pretty sure this issue is something I later encountered and fixed in #1249. If you have a chance to retry, please let me know if it works, but I'm pretty confident this was the issue so I'll go ahead and close this.