C++ library installed, but cannot be found

[SabineBellstedt]

I'm trying to install the python implementation of Profit (a bulge-disc decomposition code), called pyprofit. It can be installed via pip, but in order for this to work the C++ library libprofit needs to be installed.

I have gone through the process of installing libprofit (I thought successfully) via cmake / make, but when I go to install pyprofit I get an error:

error: No libprofit installation found on your system.

How do I point the computer to the installation it just did?

[manodeep]

Is the pyprofit library a shared library? (As in, if you go into the pyprofit directory (the highest level directory containing an __init__.py, do you see anything *.so?)

(I can tell you an easy solution but might be illustrative to learn what is going in locating this libprofit install. And even possibly suggesting a fix to the authors of pyprofit)

[SabineBellstedt]

um... Not entirely sure how to check this. Given that the pip install pyprofit is failing pretty much instantly, the install hasn't created any sort of directory yet...

[manodeep]

Ahh I mis-understand. In that case, there is nothing to do - you have to make sure that the operating system knows where libprofit is. The question is how do you tell the o/s the location of libprofit (or any other library, for that matter)

[SabineBellstedt]

The pip error is giving me instructions on how to point the install in the right direction:

You can specify a libprofit installation directory via the LIBPROFIT_HOME environment variable. Additionally, you can also use the LIBPROFIT_INCIDR and LIBPROFIT_LIBDIR environment variables to point separately to the headers and library directories respectivelly For example: LIBPROFIT_HOME=~/local python setup.py install

My problem is just that I myself don't know where libprofit has installed.....

[manodeep]

How did you install libprofit?

[SabineBellstedt]

I followed the install directions after downloading the source code:

mkdir build cd build cmake .. make sudo make install

[manodeep]

Can you re-do the sudo make install step and copy-paste the output?

(Sidenote you use three backticks, then new line, and bunch of code, and then close again with three backticks, then you will get the following)

mkdir build
cd build
cmake ..
make
sudo make install

[SabineBellstedt]

Ah, I see what you're doing here.

Install the project...
-- Install configuration: ""
-- Installing: /usr/local/lib/libprofit.dylib
-- Installing: /usr/local/bin/profit-cli
-- Installing: /usr/local/include/profit/config.h
-- Installing: /usr/local/include/profit/common.h
-- Installing: /usr/local/include/profit/convolve.h
-- Installing: /usr/local/include/profit/exceptions.h
-- Installing: /usr/local/include/profit/fft.h
-- Installing: /usr/local/include/profit/image.h
-- Installing: /usr/local/include/profit/library.h
-- Installing: /usr/local/include/profit/model.h
-- Installing: /usr/local/include/profit/opencl.h
-- Installing: /usr/local/include/profit/profile.h
-- Installing: /usr/local/include/profit/profit.h
-- Installing: /usr/local/include/profit/cl/cl2.hpp

So I take it that's the path that I just need to direct pyprofit to?

(And thanks for the tip :P)

[SabineBellstedt]

Hmm, nope it didn't seem to be as easy as that. LIBPROFIT_HOME=~/usr/local/include/profit/ sudo python setup.py install still throws the same error.

[manodeep]

Remove the initial ~ in the LIBPROFIT_HOME - that should do it.

[SabineBellstedt]

Nope, still not working without the ~. I've also tried the path /usr/local/lib/, and that doesn't do the job either.

[manodeep]

What does export LIBPROFIT_HOME=/usr/local && sudo python setup.py install do?

[SabineBellstedt]

the following:

running install
running bdist_egg
running egg_info
writing pyprofit.egg-info/PKG-INFO
writing top-level names to pyprofit.egg-info/top_level.txt
writing dependency_links to pyprofit.egg-info/dependency_links.txt
reading manifest file 'pyprofit.egg-info/SOURCES.txt'
writing manifest file 'pyprofit.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.10-intel/egg
running install_lib
running build_ext
running configure
-- Using '-std=c++11' to enable C++11 support
-- no suitable libprofit headers not found
error: No libprofit installation found on your system.

Supported versions are: 1.5.1, 1.5.2, 1.5.3, 1.6.0

You can specify a libprofit installation directory via the LIBPROFIT_HOME environment variable.
Additionally, you can also use the LIBPROFIT_INCIDR and LIBPROFIT_LIBDIR environment variables
to point separately to the headers and library directories respectivelly

For example:

LIBPROFIT_HOME=~/local python setup.py install

[manodeep]

Hmm - looking at the pyprofit setup.py, I think you may have to specify the LIBPROFIT_INCDIR and the LIBPROFIT_LIBDIR (funny that they have a typo in their error/info message).

The setup.py is looking for $LIBPROFIT_INCDIR/profit/config.h and testing that for version info (see here)

[SabineBellstedt]

That makes sense, and yet I don't understand why I can't get it to work. Have I selected the right paths in the following?

LIBPROFIT_HOME=/usr/local/ LIBPROFIT_INCDIR=/usr/local/include/ LIBPROFIT_LIBDIR=/usr/local/lib/ sudo python setup.py install

[manodeep]

LIBPROFIT_INCIDR should be LIBPROFIT_INCDIR (the typo that I just fixed with the pull request)

(Might take some staring to notice the difference)

[SabineBellstedt]

Yeah I noticed that second typo just after I posted the comment! Even fixed, the install still isn't working...

[manodeep]

Can you add some debug print messages into the setup.py in this section

As in print out the full filename, and if the file is not found etc. And check manually if the $LIBPROFIT_INCDIR/profit/config.h file actually exists on your system. Definitely print out the value of the variable $LIBPROFIT_INCDIR as seen with the setup.py

[SabineBellstedt]

My debugging as shown that all of the environments seem to be completely undefined, and my usage of environment variables hasn't passed any information to the code. the LIBPROFIT_INCDIR variable as well as the others are all None, which is simply the default value that is passed in...

Thanks for helping me with this Manodeep!

[manodeep]

Suspected as much. Does adding export LIBPROFIT_HOME=/usr/local/ change anything?

[SabineBellstedt]

You mean to the call, or the code itself?

[manodeep]

On the shell command, doesn't need to be on the same line

export LIBPROFIT_HOME=/usr/local/
export LIBPROFIT_INCDIR=/usr/local/include/ 
export LIBPROFIT_LIBDIR=/usr/local/lib/ 
sudo python setup.py install

[SabineBellstedt]

Nope, same outcome as before.

[manodeep]

Your shell is bash - yes?

[SabineBellstedt]

It is.

[manodeep]

Then I don't know how the setup.py ever worked for anyone that had a custom install. You can probably fix it for yourself by checking within setup.py if those environment variables are undefined, and then defining them to the values you want.

[manodeep]

Would be nice to have a real solution that you could then contribute back to the pyprofit repo

[SabineBellstedt]

Okay, that I can do! Thanks a lot :)

[SabineBellstedt]

Ah, one more thing I clearly don't understand: what is the purpose of the compiler_incdirs parameter - as in, what should it be? It's also currently empty, even if I manually define the input variables...

[SabineBellstedt]

Ah, I may have just figured it out. The libprofit version I installed is 1.7.0, which is still in development, and not technically one of the supported versions...

[manodeep]

Will you set export DISTUTILS_DEBUG=1 and then re-run the python setup.py. That should display the log file from the install.

The setup.py definitely checks for the version. But given that none of the environment variables were being captured, I am not sure how that is working

[manodeep]

I am confused that the os.environ.get statement is not picking up the variables.

Before you run the setup command, will you do echo $LIBPROFIT_HOME and check the environment variable is actually set. If so, then startup python and then the following

>>> from __future__ import print_function
>>> import os
>>> print(os.environ.get('LIBPROFIT_HOME', None))

[SabineBellstedt]

Yeah, that's all working perfectly in the terminal! So I'm not sure why the os.environ.get doesn't work either

[SabineBellstedt]

Also the export DISTUTILS_DEBUG=1 doesn't seem to change the python setup.py output at all..

[manodeep]

Perhaps change the DISTUTILS_DEBUG=2 to get the info messages and retry

[SabineBellstedt]

nope still the same..??

[manodeep]

Or run with python -v

[manodeep]

(huh, there is a race condition in the github comments. I clicked comment and you did too - but my comment has disappeared)

The comment was try with python -v

[SabineBellstedt]

ah that did it. What are you looking for specifically? The output is longer than my terminal history...

[manodeep]

Never mind...I see the comment now. :D

[manodeep]

Save it into a log file sudo python -v setup.py install &> install.log

[manodeep]

For the distutils issue, this claims that setting distutils.log.set_verbosity(1) after the import distutils in the setup.py should work.

[SabineBellstedt]

Do you want to see the full output? It includes a bunch of input messages, followed by the original error message, then a bunch of cleanup messages.

[SabineBellstedt]

# installing zipimport hook
import zipimport # builtin
# installed zipimport hook
# /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site.pyc matches /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site.py
import site # precompiled from /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site.pyc
# /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.pyc matches /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.py
import os # precompiled from /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.pyc
import errno # builtin
import posix # builtin
# /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.pyc matches /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.py
import posixpath # precompiled from /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/posixpath.pyc
# /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/stat.pyc matches /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/stat.py
import stat # precompiled from /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/stat.pyc
# /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/genericpath.pyc matches /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/genericpath.py
import genericpath # precompiled from /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/genericpath.pyc
# /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/warnings.pyc matches /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/warnings.py
import warnings # precompiled from /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/warnings.pyc
# /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/linecache.pyc matches /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/linecache.py
import linecache # precompiled from /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/linecache.pyc
# /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/types.pyc matches /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/types.py
import types # precompiled from /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/types.pyc
# /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/UserDict.pyc matches /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/UserDict.py
import UserDict # precompiled from /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/UserDict.pyc
# /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_abcoll.pyc matches /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_abcoll.py
import _abcoll # precompiled from /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_abcoll.pyc
# /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/abc.pyc matches /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/abc.py
import abc # precompiled from /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/abc.pyc
# /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_weakrefset.pyc matches /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_weakrefset.py
import _weakrefset # precompiled from /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_weakrefset.pyc
import _weakref # builtin
# /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy_reg.pyc matches /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy_reg.py
import copy_reg # precompiled from /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy_reg.pyc
# /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/traceback.pyc matches /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/traceback.py
import traceback # precompiled from /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/traceback.pyc
# /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sysconfig.pyc matches /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sysconfig.py
import sysconfig # precompiled from /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sysconfig.pyc
# /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/re.pyc matches /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/re.py
import re # precompiled from /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/re.pyc
# /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sre_compile.pyc matches /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sre_compile.py
import sre_compile # precompiled from /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sre_compile.pyc
import _sre # builtin
# /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sre_parse.pyc matches /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sre_parse.py
import sre_parse # precompiled from /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sre_parse.pyc
# /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sre_constants.pyc matches /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sre_constants.py
import sre_constants # precompiled from /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/sre_constants.pyc
# /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_sysconfigdata.pyc matches /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_sysconfigdata.py
import _sysconfigdata # precompiled from /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_sysconfigdata.pyc
# /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_osx_support.pyc matches /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_osx_support.py
import _osx_support # precompiled from /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/_osx_support.pyc
# zipimport: found 10 names in /Library/Python/2.7/site-packages/linmix-0.1.0.dev1-py2.7.egg
# zipimport: found 11 names in /Library/Python/2.7/site-packages/progressbar-2.3-py2.7.egg
# zipimport: found 27 names in /Library/Python/2.7/site-packages/pywcsgrid2-1.0_git-py2.7.egg
# zipimport: found 123 names in /Library/Python/2.7/site-packages/setuptools-25.1.6-py2.7.egg
import encodings # directory /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings
# /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/__init__.pyc matches /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/__init__.py
import encodings # precompiled from /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/__init__.pyc
# /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/codecs.pyc matches /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/codecs.py
import codecs # precompiled from /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/codecs.pyc
import _codecs # builtin
# /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/aliases.pyc matches /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/aliases.py
import encodings.aliases # precompiled from /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/aliases.pyc
# /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/utf_8.pyc matches /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/utf_8.py
import encodings.utf_8 # precompiled from /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/utf_8.pyc
Python 2.7.6 (default, Sep  9 2014, 15:04:36) 
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
import distutils # directory /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils
# /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/__init__.pyc matches /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/__init__.py
import distutils # precompiled from /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/__init__.pyc
Traceback (most recent call last):
  File "setup.py", line 23, in <module>
    distutils.log.set_verbosity(1)
AttributeError: 'module' object has no attribute 'log'
# clear __builtin__._
# clear sys.path
# clear sys.argv
# clear sys.ps1
# clear sys.ps2
# clear sys.exitfunc
# clear sys.exc_type
# clear sys.exc_value
# clear sys.exc_traceback
# clear sys.last_type
# clear sys.last_value
# clear sys.last_traceback
# clear sys.path_hooks
# clear sys.path_importer_cache
# clear sys.meta_path
# clear sys.flags
# clear sys.float_info
# restore sys.stdin
# restore sys.stdout
# restore sys.stderr
# cleanup __main__
# cleanup[1] distutils
# cleanup[1] encodings
# cleanup[1] site
# cleanup[1] sysconfig
# cleanup[1] abc
# cleanup[1] _weakrefset
# cleanup[1] sre_constants
# cleanup[1] _codecs
# cleanup[1] _warnings
# cleanup[1] zipimport
# cleanup[1] _sysconfigdata
# cleanup[1] mpl_toolkits
# cleanup[1] encodings.utf_8
# cleanup[1] _osx_support
# cleanup[1] codecs
# cleanup[1] signal
# cleanup[1] traceback
# cleanup[1] posix
# cleanup[1] encodings.aliases
# cleanup[1] exceptions
# cleanup[1] _weakref
# cleanup[1] re
# cleanup[1] sre_compile
# cleanup[1] _sre
# cleanup[1] sre_parse
# cleanup[2] copy_reg
# cleanup[2] posixpath
# cleanup[2] errno
# cleanup[2] _abcoll
# cleanup[2] types
# cleanup[2] genericpath
# cleanup[2] stat
# cleanup[2] warnings
# cleanup[2] UserDict
# cleanup[2] os.path
# cleanup[2] linecache
# cleanup[2] os
# cleanup sys
# cleanup __builtin__
# cleanup ints: 1017 unfreed ints
# cleanup floats

[SabineBellstedt]

oops, that one has an error. The output without setting distutils.log.set_verbosity(1) is much longer...

[manodeep]

Why is that error occurring?! The same error should occur with python/pip while trying to install then

[manodeep]

With the distutils.log.set_verbosity, you don't need the -v flag (in theory)

[SabineBellstedt]

Yeah the error just came because I didn't put it late enough after all the distutils imports, unsurprisingly. Although it looks like the -v flag was still required the get the full output. Did you want to see it, even though it's huge?

[manodeep]

I just want to see the output from distutils. But attach the whole log file anyway....

[SabineBellstedt]

install.log