Problems with pybertini install

[jbcolli2]

After compiling and installing both the core code and pybertini code, a couple of issues arose. I am running on a Macbook pro.

1. The pybertini module could not be found from the test subdirectory. This issue could be due to multiple versions of python on my computer. I have installed python via brew and anaconda, and so the folder where the _pybertini.so file was installed was not part of my PYTHONPATH.

2. Once the PYTHONPATH was fixed, the _pybertini module could be found, but the pybertini module could not be found from the test subdirectory.

3. If the _pybertini module was imported, a segmentation fault was returned.

[ofloveandhate]

on 1)., that does not surprise me, though it is something we should address. i had to add this to my env:

PYTHONPATH=:/usr/local/lib/python3.6/site-packages

because that's where it installs. we should print a message indicating that this is necessary.

on 2)., the boost.python bindings were renamed from pybertini to _pybertini, explaining this one. the missing step is pip install ., or if it was already installed, pip install --upgrade . from b2/python. this step should be integrated into the make install routine.

on 3), that's a new one for me. my guess is an ABI mismatch. did you recompile the core and reinstall it before compiling pybertini?

[jbcolli2]

Ok, the pip install . fixed problem 2). As for 3), I did reinstall the core before compiling pybertini. I just did it all again and still run into the same issue. Could it be a problem with my two python distributions installed on my machine?

[ofloveandhate]

yep. probably python 2 and 3 fighting. can you check which pybertini was compiled against? it should be in config.log.

[jbcolli2]

Yeah, it's not python 2 and 3, but my two versions of python 3 that are fighting.

So I installed Anaconda, with all the python that goes with it, and then I installed python3 using brew. I think I needed pip3, which was why I installed it.

Now configure is finding both of those versions and using them both for different things. The pybertini module is being installed in the Anaconda site_package, but the _pybertini is being installed in the usr/local/lib/... subdirectory.

It also looks like the python used for compiling is both of them. There is a flag in config.log

PYTHON = '/Applications/Programming/Anaconda/anaconda3/bin/python'

but also

PYTHON_EXEC_PREFIX= '/usr/local'

So somehow it seems to be using both versions of python on my machine.

[ofloveandhate]

here is the portion of my config.log:

configure:19220: checking for python3.6
configure:19250: result: /Users/brakeda/anaconda/bin/python
configure:19266: checking for a version of Python >= '2.1.0'
configure:19292: result: yes
configure:19323: checking for the distutils Python package
configure:19327: result: yes
configure:19341: checking for Python include path
configure:19357: result: -I/Users/brakeda/anaconda/include/python3.6m
configure:19364: checking for Python library path
configure:19441: result: -L/Users/brakeda/anaconda/lib -lpython3.6m
configure:19448: checking for Python site-packages path
configure:19454: result: /Users/brakeda/anaconda/lib/python3.6/site-packages
configure:19461: checking python extra libraries
configure:19468: result: -ldl  -framework CoreFoundation 
configure:19475: checking python extra linking flags
configure:19482: result: -Wl,-stack_size,1000000  -framework CoreFoundation
configure:19489: checking consistency of all components of python development environment
configure:19517: gcc -o conftest -g -O2  -I/usr/local/include/eigen3 -I/Users/brakeda/anaconda/include/python3.6m  -ldl  -framework CoreFoundation  conftest.c -lmpfr -lgmp  -L/Users/brakeda/anaconda/lib -lpython3.6m -Wl,-stack_size,1000000  -framework CoreFoundation -Wl,-stack_size,1000000  -framework CoreFoundation >&5
configure:19517: $? = 0
configure:19535: result: yes
configure:19591: checking whether the Boost::Python library is available
configure:19609: g++ -c -g -O2 -std=gnu++14 -pthread -I/usr/local/include -I/Users/brakeda/anaconda/include/python3.6m  -I/usr/local/include/eigen3 conftest.cpp >&5
configure:19609: $? = 0
configure:19617: result: yes
configure:19636: checking whether boost_python is the correct library
configure:19655: g++ -o conftest -g -O2 -std=gnu++14 -pthread -I/usr/local/include -I/Users/brakeda/anaconda/include/python3.6m  -I/usr/local/include/eigen3 -L/usr/local/lib  conftest.cpp -lboost_python -lmpfr -lgmp  -L/Users/brakeda/anaconda/lib -lpython3.6m >&5
ld: library not found for -lboost_python
clang: error: linker command failed with exit code 1 (use -v to see invocation)
configure:19655: $? = 1
configure: failed program was:
| /* confdefs.h */
| #define PACKAGE_NAME "pybertini"
| #define PACKAGE_TARNAME "pybertini"
| #define PACKAGE_VERSION "1.0.alpha2"
| #define PACKAGE_STRING "pybertini 1.0.alpha2"
| #define PACKAGE_BUGREPORT "brakeda@uwec.edu"
| #define PACKAGE_URL "http://github.com/bertiniteam/b2"
| #define PACKAGE "pybertini"
| #define VERSION "1.0.alpha2"
| #define STDC_HEADERS 1
| #define HAVE_SYS_TYPES_H 1
| #define HAVE_SYS_STAT_H 1
| #define HAVE_STDLIB_H 1
| #define HAVE_STRING_H 1
| #define HAVE_MEMORY_H 1
| #define HAVE_STRINGS_H 1
| #define HAVE_INTTYPES_H 1
| #define HAVE_STDINT_H 1
| #define HAVE_UNISTD_H 1
| #define HAVE_DLFCN_H 1
| #define LT_OBJDIR ".libs/"
| #define HAVE_EIGEN_DENSE 1
| #define HAVE_BOOST /**/
| #define HAVE_BOOST_SYSTEM /**/
| #define HAVE_BOOST_FILESYSTEM /**/
| #define HAVE_BOOST_CHRONO /**/
| #define HAVE_BOOST_REGEX /**/
| #define HAVE_BOOST_TIMER /**/
| #define HAVE_BOOST_UNIT_TEST_FRAMEWORK /**/
| #define HAVE_BOOST_SERIALIZATION /**/
| #define HAVE_BOOST_LOG /**/
| #define HAVE_BOOST_LOG_SETUP /**/
| #define HAVE_BOOST_THREAD /**/
| #define HAVE_PYTHON "3.6"
| #define HAVE_BOOST_PYTHON /**/
| /* end confdefs.h.  */
| 
| #include <boost/python/module.hpp>
| BOOST_PYTHON_MODULE(test) { throw "Boost::Python test."; }
| int
| main ()
| {
| 
|   ;
|   return 0;
| }
configure:19664: result: no
configure:19636: checking whether boost_python3-mt is the correct library
configure:19655: g++ -o conftest -g -O2 -std=gnu++14 -pthread -I/usr/local/include -I/Users/brakeda/anaconda/include/python3.6m  -I/usr/local/include/eigen3 -L/usr/local/lib  conftest.cpp -lboost_python3-mt -lmpfr -lgmp  -L/Users/brakeda/anaconda/lib -lpython3.6m >&5
configure:19655: $? = 0
configure:19664: result: yes

so it picked up the "right" place to install things. i think that it bases much of its search on the value of

configure:19250: result: /Users/brakeda/anaconda/bin/python

here is the bottom portion of my config.log:

PYTHON='/Users/brakeda/anaconda/bin/python'
PYTHON_CPPFLAGS='-I/Users/brakeda/anaconda/include/python3.6m'
PYTHON_EXEC_PREFIX='${exec_prefix}'
PYTHON_EXTRA_LDFLAGS='-ldl  -framework CoreFoundation '
PYTHON_EXTRA_LIBS='-Wl,-stack_size,1000000  -framework CoreFoundation'
PYTHON_LIBS='-L/Users/brakeda/anaconda/lib -lpython3.6m'
PYTHON_PLATFORM='darwin'
PYTHON_PREFIX='${prefix}'
PYTHON_SITE_PKG='/Users/brakeda/anaconda/lib/python3.6/site-packages'
PYTHON_VERSION='3.6'

so, i think that having multiple versions of python3 around is confusing your setup.

[ofloveandhate]

it is possible that the version of python that boost.python is linked to is not compatible with the python lib it found. will you check the value of BOOST_PYTHON_LIB in your b2/python/config.log?

[ofloveandhate]

i had to install boost.python through brew using additional flags to get things to work for me.

[ofloveandhate]

brew install boost-python --with-python3

[jbcolli2]

I have BOOST_PYTHON_LIB='-lboost_python3-mt'

What do you see as the value of exec_prefix and prefix? That is where the other python distribution seems to be showing up for me.

[ofloveandhate]

PYTHON='/Users/brakeda/anaconda/bin/python'
PYTHON_CPPFLAGS='-I/Users/brakeda/anaconda/include/python3.6m'
PYTHON_EXEC_PREFIX='${exec_prefix}'
PYTHON_EXTRA_LDFLAGS='-ldl  -framework CoreFoundation '
PYTHON_EXTRA_LIBS='-Wl,-stack_size,1000000  -framework CoreFoundation'
PYTHON_LIBS='-L/Users/brakeda/anaconda/lib -lpython3.6m'
PYTHON_PLATFORM='darwin'
PYTHON_PREFIX='${prefix}'
PYTHON_SITE_PKG='/Users/brakeda/anaconda/lib/python3.6/site-packages'
PYTHON_VERSION='3.6'

[jbcolli2]

Sorry, I meant the actual ${exec_prefix} and ${prefix}.

[ofloveandhate]

exec_prefix='${prefix}'
prefix='/usr/local'

[ofloveandhate]

here is my entire config.log config.log

[ofloveandhate]

something you could try, that probably won't make a difference. autoreconfing, then re-configuring. maybe there are old versions of the m4 macros hanging out in your configure script?

[jbcolli2]

Issue is not resolved, but understood better.

The problem occurs because of some conflict between Anaconda boost-python3. The boost-python3 was installed using brew. When python is run through Anaconda, a Segmentation Fault occurs, possibly because pybertini was compiled using boost-python3. No Segmentation Fault occurs when boost-python is used for compiling. The version of boost is not an issue, nor is the version of conda.

[ofloveandhate]

check out this issue for roboschool

[ofloveandhate]

so, i was able to get it going on my laptop after experiencing segfaults and then a messed up @rpath/.../libpython3.6m or something, by uninstalling the python and python3 provided by homebrew, uninstalling boost-python, and re-installing boost-python.

bottom line -- anaconda / conda and brew don't play nicely when it comes to python. if you have both installed and visible, then it's easy to get errors when import pybertini is called, from whatever source.

jeb, can you try the above, and see if it works? also, i added #143 to document the need for containerizing for distribution.

[ofloveandhate]

here's another bit of reading to consider:

https://hashrocket.com/blog/posts/keep-anaconda-from-constricting-your-homebrew-installs

[ofloveandhate]

somehow i got pybertini to import correctly in anaconda python on my laptop, but have been unsuccessful on my desktop....

[jbcolli2]

I gave all that a try, uninstalled boost-python3, python3 with brew (I didn't have boost-python installed). I reinstalled boost-python --with-python3. I ran autoreconf and configure for the python code and ran into the following error on configure

checking whether the Boost::Python library is available... yes checking whether boost_python is the correct library... no checking whether boost_python-mt is the correct library... no checking whether boost_python-mt is the correct library... (cached) no checking whether boost_python is the correct library... (cached) no checking whether boost_python is the correct library... (cached) no checking whether boost_python is the correct library... (cached) no checking whether boost_python3 is the correct library... no configure: error: Failed to find correct version of Boost.Python!

Any ideas?

[ofloveandhate]

take anaconda off the path both to configure, compile, and use pybertini. otherwise, perhaps boost.python can be compiled standalone against brew-installed boost, and the anaconda python. anaconda and brew do not play nicely with each other. this is why i want to containerize pybertini if possible.

[ofloveandhate]

also, it appears that to get python3-compatible boost.python, with brew you need to brew install boost-python3. this is a most aggravating problem, but i guess what you get when multiple package managers are fighting for control.

[jbcolli2]

I have removed Anaconda from my path. I then installed boost-python --with-python3, with boost-python3 uninstalled. I then reconfigured, ran make clean, then compiled the python libraries. When I try to import pybertini when running my computer's python, I get the following message:

ImportError: No module named _pybertini.system

I then do the same think with python3. I uninstall boost-python and install boost-python3. I then reconfigure. During configure I get:

checking whether the Boost::Python library is available... yes
checking whether boost_python is the correct library... no
checking whether boost_python3-mt is the correct library... no
checking whether boost_python3-mt is the correct library... (cached) no
checking whether boost_python3 is the correct library... no
checking whether boost_python3 is the correct library... (cached) no
checking whether boost_python is the correct library... (cached) no
checking whether boost_python3 is the correct library... (cached) no
configure: error: Failed to find correct version of Boost.Python!

I then brew install boost-python --with-python3 and the configure works. I clean, compile, and when I run the brewed python3 and import pybertini I get:

ImportError: dlopen(/usr/local/lib/python3.6/site-packages/_pybertini.so, 2): Library not loaded: @rpath/libpython3.6m.dylib
  Referenced from: /usr/local/lib/python3.6/site-packages/_pybertini.so
  Reason: image not found

Both of these errors happen when _pybertini.system is imported.

[ofloveandhate]

• did you make install?
• is the location of the _pybertini.so install on your PYTHON_PATH?

[jbcolli2]

Yes, the PYTHONPATH was missing the python2.7 site-packages folder. Now I can import pybertini in python 2.7.

A couple of things,

1. When running the tests, it looks like there is an error with the endgame tests

   File "/Users/jebcollins/Documents/Research/Bertini/JebsFork_b2/python/test/tracking/endgame_test.py", line 42, in <module>
   from pybertini.endgame import *
   AttributeError: 'module' object has no attribute '__loader__'

   Is this to be expected?

2. The pybertini module, not the _pybertini, is not copied into a usr/local/ type place, it is just in my local directory with the rest of the code. Is it supposed to be copied into a site-packages directory, or is it doing the correct behavior?

3. When configuring, it seems to find my python 2.7 before python 3.6. My guess is because python points to 2.7 while python3 points to 3.6. Is there a fix for having configure find my python3.6 first?

[ofloveandhate]

re:

When configuring, it seems to find my python 2.7 before python 3.6. My guess is because python points to 2.7 while python3 points to 3.6. Is there a fix for having configure find my python3.6 first?

reading ./configure --help, i see the following:

Some influential environment variables:
  CXX         C++ compiler command
  CXXFLAGS    C++ compiler flags
  LDFLAGS     linker flags, e.g. -L<lib dir> if you have libraries in a
              nonstandard directory <lib dir>
  LIBS        libraries to pass to the linker, e.g. -l<library>
  CPPFLAGS    (Objective) C/C++ preprocessor flags, e.g. -I<include dir> if
              you have headers in a nonstandard directory <include dir>
  CC          C compiler command
  CFLAGS      C compiler flags
  LT_SYS_LIBRARY_PATH
              User-defined run-time library search path.
  CPP         C preprocessor
  CXXCPP      C++ preprocessor
  PYTHON      the Python interpreter
  PYTHON_VERSION
              The installed Python version to use, for example '2.3'. This
              string will be appended to the Python interpreter canonical
              name.

so, i think that ./configure PYTHON=python3 PYTHON_VERSION=3.6 is probably the ticket. i think i use this on my build server

[ofloveandhate]

re:

The pybertini module, not the _pybertini, is not copied into a usr/local/ type place, it is just in my local directory with the rest of the code. Is it supposed to be copied into a site-packages directory, or is it doing the correct behavior?

pip install . --upgrade

from b2/python, possibly using --user or something if permissions don't allow a global install.

re:

When running the tests, it looks like there is an error with the endgame tests [snip]

i believe installing pybertini through pip will solve the problem

[jbcolli2]

Ok, everything seems to be working now. I'm missing the ease of Anaconda, but I can still get spyder through pip.

[ofloveandhate]

i agree that anaconda is useful in its ease of use. i don't want to lose users of bertini2 / pybertini due to anaconda conflicts, so a solution is still very desirable.

glad you are at least up and going again. sorry anaconda didn't work out.