search

bertiniteam / b2

Bertini 2.0: The redevelopment of Bertini in C++.
92 stars 34 forks source link

Compiling against Python3 fails #55

Closed cthoyt closed 7 years ago

cthoyt commented 8 years ago

I'm copying this issue from the discussion in issue #52 about python documentation. I'm running ./configure PYTHON=python3 and here's the error I'm getting:

checking consistency of all components of python development environment... no
configure: error: in `/Users/cthoyt/dev/b2/python':
configure: error: 
  Could not link test program to Python. Maybe the main Python library has been
  installed in some non-standard library path. If so, pass it to configure,
  via the LIBS environment variable.
  Example: ./configure LIBS="-L/usr/non-standard-path/python/lib"
  ============================================================================
   ERROR!
   You probably have to install the development version of the Python package
   for your distribution.  The exact name of this package varies among them.
  ============================================================================

Other variations, like ./configure PYTHON=python3 PYTHON_VERSION=3 and many exotic locations for LIBS haven't worked. Here's a bit more about my environment:

$ python3-config --exec
/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5
$ python3-config --prefix
/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5
$ python3-config --exec-prefix
/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5
$ python3-config --libs
-lpython3.5m -ldl -framework CoreFoundation
$ python3-config --ldflags
-L/usr/local/Cellar/python3/3.5.2_1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/config-3.5m -lpython3.5m -ldl -framework CoreFoundation
jacobbond commented 8 years ago

You said you tried a bunch of variations, so maybe you already tried this, but it sounded to me from the configure help that configure PYTHON_VERSION=3 was the way to go. This is just a guess based on the configure help, but I think configure PYTHON=python3 PYTHON_VERSION=3 will cause it to try to use python33.

cthoyt commented 8 years ago

With just configure PYTHON_VERSION=3 it still installed to the default python which is Python2.7. I also tried configure PYTHON_VERSION=3.5 and configure PYTHON_VERSION=35 and got the same result.

ofloveandhate commented 8 years ago

will you please check which version of Python your Boost.Python is built against?

Also, another great place to look and see what exactly failed is the file config.log. It looks like tons of stuff failing, because the pattern is something like

  1. write simple c++ program
  2. test compiling using a specific invocation of compiler
  3. do something on success of compiling, maybe goto 1 or goto 2, or set a variable or add something to LDFLAGS, etc.

so, there are tons of failed compiles in there. that's normal. it's how ./configure works.

the bit you are interested in is probably the last one or two before the summary of cached variables. when i try to diagnose these problems using config.log, i go to the end of the file, and scroll up until i see some compiler results. those are probably the ones i am interested in. also, feel free to attach files like config.log to the issue here.

the code for finding python is entirely from the autoconf archive, so if there are bugs, i should look and see if there is a new version of the m4 macros. but i kinda doubt there are problems -- it's more likely to be an environment problem. mac comes with python, and @cthoyt has two Homebrew-installed python versions, 2 and 3, if i remember correctly. so making sure that Boost.Python used the correct one when it was built is probably the primary concern.

ofloveandhate commented 7 years ago

so i'm on El Capitan with homebrew for this writing, November 9, 2016.

I ran brew install python3 to install python3, and I have the brew-supplied versions of Boost and Boost.Python, right now at 1.62 and 1.61 respectively. ./configure PYTHON=python3 ran kinda ok for me, with this output:

integral:python ofloveandhate$ ./configure PYTHON=python3
checking for ./minieigen/src/common.hpp... yes
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... config/install-sh -c -d
checking for gawk... no
checking for mawk... no
checking for nawk... no
checking for awk... awk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for style of include used by make... GNU
checking for g++... g++
checking whether the C++ compiler works... yes
checking for C++ compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C++ compiler... yes
checking whether g++ accepts -g... yes
checking dependency style of g++... gcc3
checking whether g++ supports C++14 features by default... no
checking whether g++ supports C++14 features with -std=gnu++14... yes
checking whether ln -s works... yes
checking build system type... x86_64-apple-darwin15.6.0
checking host system type... x86_64-apple-darwin15.6.0
checking how to print strings... printf
checking for gcc... gcc
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether gcc understands -c and -o together... yes
checking dependency style of gcc... gcc3
checking for a sed that does not truncate output... /usr/bin/sed
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for fgrep... /usr/bin/grep -F
checking for ld used by gcc... /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld
checking if the linker (/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld) is GNU ld... no
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking the maximum length of command line arguments... 196608
checking how to convert x86_64-apple-darwin15.6.0 file names to x86_64-apple-darwin15.6.0 format... func_convert_file_noop
checking how to convert x86_64-apple-darwin15.6.0 file names to toolchain format... func_convert_file_noop
checking for /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld option to reload object files... -r
checking for objdump... no
checking how to recognize dependent libraries... pass_all
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for ar... ar
checking for archiver @FILE support... no
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for sysroot... no
checking for a working dd... /bin/dd
checking how to truncate binary pipes... /bin/dd bs=4096 count=1
checking for mt... no
checking if : is a manifest tool... no
checking for dsymutil... dsymutil
checking for nmedit... nmedit
checking for lipo... lipo
checking for otool... otool
checking for otool64... no
checking for -single_module linker flag... yes
checking for -exported_symbols_list linker flag... yes
checking for -force_load linker flag... yes
checking how to run the C preprocessor... gcc -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... yes
checking for gcc option to produce PIC... -fno-common -DPIC
checking if gcc PIC flag -fno-common -DPIC works... yes
checking if gcc static flag -static works... no
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld) supports shared libraries... yes
checking dynamic linker characteristics... darwin15.6.0 dyld
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking how to run the C++ preprocessor... g++ -E
checking for ld used by g++... /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld
checking if the linker (/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld) is GNU ld... no
checking whether the g++ linker (/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld) supports shared libraries... yes
checking for g++ option to produce PIC... -fno-common -DPIC
checking if g++ PIC flag -fno-common -DPIC works... yes
checking if g++ static flag -static works... no
checking if g++ supports -c -o file.o... yes
checking if g++ supports -c -o file.o... (cached) yes
checking whether the g++ linker (/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld) supports shared libraries... yes
checking dynamic linker characteristics... darwin15.6.0 dyld
checking how to hardcode library paths into programs... immediate
checking for python version... 3.5
checking for python platform... darwin
checking for python script directory... ${prefix}/lib/python3.5/site-packages
checking for python extension module directory... ${exec_prefix}/lib/python3.5/site-packages
checking for library containing cos... none required
checking for library containing __gmpz_init... -lgmp
checking for library containing mpfr_get_version... -lmpfr
checking Eigen/Dense usability... yes
checking Eigen/Dense presence... yes
checking for Eigen/Dense... yes
checking for boostlib >= 1.58... yes
checking whether the Boost::System library is available... yes
checking for exit in -lboost_system... yes
checking whether the Boost::Filesystem library is available... yes
checking for exit in -lboost_filesystem... yes
checking whether the Boost::Chrono library is available... yes
checking for exit in -lboost_chrono-mt... yes
checking whether the Boost::Regex library is available... yes
checking for exit in -lboost_regex-mt... yes
checking whether the Boost::Timer library is available... yes
checking for exit in -lboost_timer-mt... yes
checking whether the Boost::Unit_Test_Framework library is available... yes
checking whether the Boost::Serialization library is available... yes
checking for exit in -lboost_serialization-mt... yes
checking whether the Boost::Log library is available... yes
checking for exit in -lboost_log-mt... yes
checking for exit in -lboost_log_setup-mt... yes
checking whether the Boost::Thread library is available... yes
checking for exit in -lboost_thread-mt... yes
checking for python3.5... /usr/local/bin/python3.5
checking for a version of Python >= '2.1.0'... yes
checking for the distutils Python package... yes
checking for Python include path... -I/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/include/python3.5m
checking for Python library path... -L/usr/local/opt/python3/Frameworks/Python.framework/Versions/3.5/lib -lpython3.5m
checking for Python site-packages path... /usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages
checking python extra libraries... -ldl  -framework CoreFoundation 
checking python extra linking flags... -Wl,-stack_size,1000000  -framework CoreFoundation /usr/local/opt/python3/Frameworks/Python.framework/Versions/3.5/Python
checking consistency of all components of python development environment... yes
checking whether the Boost::Python library is available... yes
checking whether boost_python is the correct library... no
checking whether boost_python-mt is the correct library... no
checking whether boost_python-mt is the correct library... (cached) no
checking whether boost_python is the correct library... (cached) no
checking whether boost_python is the correct library... (cached) no
checking whether boost_python is the correct library... (cached) no
checking whether boost_python3 is the correct library... no
checking bertini2/mpfr_complex.hpp usability... yes
checking bertini2/mpfr_complex.hpp presence... yes
checking for bertini2/mpfr_complex.hpp... yes
checking for library containing HaveBertini2... -lbertini2
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating config.h
config.status: executing depfiles commands
config.status: executing libtool commands

i then proceeded to make (in parallel, using tons of ram as usual), and got to linking, where mine died, saying 

/bin/sh ./libtool  --tag=CXX   --mode=link g++ -I/usr/local/Cellar/python3/3.5.2_3/Frameworks/Python.framework/Versions/3.5/include/python3.5m -pthread -I/usr/local/include -I/usr/local/include -g -O2 -std=gnu++14 -module -avoid-version -shared  -o pybertini.la -rpath /usr/local/lib/python3.5/site-packages  src/pybertini_la-bertini_python.lo src/pybertini_la-tracker_export.lo src/pybertini_la-endgame_export.lo src/pybertini_la-mpfr_export.lo src/pybertini_la-node_export.lo src/pybertini_la-symbol_export.lo src/pybertini_la-operator_export.lo src/pybertini_la-root_export.lo src/pybertini_la-system_export.lo minieigen/src/pybertini_la-expose-converters.lo minieigen/src/double-conversion/pybertini_la-bignum-dtoa.lo minieigen/src/double-conversion/pybertini_la-bignum.lo minieigen/src/double-conversion/pybertini_la-cached-powers.lo minieigen/src/double-conversion/pybertini_la-diy-fp.lo minieigen/src/double-conversion/pybertini_la-double-conversion.lo minieigen/src/double-conversion/pybertini_la-fast-dtoa.lo minieigen/src/double-conversion/pybertini_la-fixed-dtoa.lo minieigen/src/double-conversion/pybertini_la-strtod.lo -L/usr/local/lib  -Wl,-stack_size,1000000  -framework CoreFoundation /usr/local/opt/python3/Frameworks/Python.framework/Versions/3.5/Python  -lboost_filesystem -lboost_system  -lboost_chrono-mt -lboost_regex-mt -lboost_timer-mt  -lboost_serialization-mt -lboost_log-mt -lboost_log_setup-mt -lboost_thread-mt -L/usr/local/lib -lbertini2 -lmpfr -lgmp 
libtool: link: rm -fr  .libs/pybertini.la .libs/pybertini.lai .libs/pybertini.so
libtool: link: g++ -Wl,-undefined -Wl,dynamic_lookup -o .libs/pybertini.so -bundle  src/.libs/pybertini_la-bertini_python.o src/.libs/pybertini_la-tracker_export.o src/.libs/pybertini_la-endgame_export.o src/.libs/pybertini_la-mpfr_export.o src/.libs/pybertini_la-node_export.o src/.libs/pybertini_la-symbol_export.o src/.libs/pybertini_la-operator_export.o src/.libs/pybertini_la-root_export.o src/.libs/pybertini_la-system_export.o minieigen/src/.libs/pybertini_la-expose-converters.o minieigen/src/double-conversion/.libs/pybertini_la-bignum-dtoa.o minieigen/src/double-conversion/.libs/pybertini_la-bignum.o minieigen/src/double-conversion/.libs/pybertini_la-cached-powers.o minieigen/src/double-conversion/.libs/pybertini_la-diy-fp.o minieigen/src/double-conversion/.libs/pybertini_la-double-conversion.o minieigen/src/double-conversion/.libs/pybertini_la-fast-dtoa.o minieigen/src/double-conversion/.libs/pybertini_la-fixed-dtoa.o minieigen/src/double-conversion/.libs/pybertini_la-strtod.o   -L/usr/local/lib -framework CoreFoundation /usr/local/lib/libbertini2.dylib -lboost_serialization-mt -lboost_system -lboost_filesystem -lboost_chrono-mt -lboost_regex-mt -lboost_thread-mt -lboost_timer-mt -lboost_log-mt -lboost_log_setup-mt -lmpfr -lgmp  -pthread -g -O2 -Wl,-stack_size -Wl,1000000   -pthread -framework CoreFoundation
clang: warning: argument unused during compilation: '-pthread'
clang: warning: argument unused during compilation: '-pthread'
ld: -stack_size option can only be used when linking a main executable
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [pybertini.la] Error 1
make: *** [all] Error 2

so my computer setup made it past the configure stage, but failed to finally link the objects into the target library, apparently due to -stack_size coming from somewhere.

sorry, i couldn't immediately replicate your configure error.

ofloveandhate commented 7 years ago

this -stack_size problem was due to $(PYTHON_EXTRA_LIBS), which i don't think I should have been using. eliminated in 30c26c393cc34772276f3a697ebfeef18c5a5a98

ofloveandhate commented 7 years ago

fixing this compilation error the yields the following runtime error:

integral:~ ofloveandhate$ python3
Python 3.5.2 (default, Oct 11 2016, 05:05:28) 
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.38)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from pybertini import *
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dlopen(/usr/local/lib/python3.5/site-packages/pybertini.so, 2): Symbol not found: _PyInt_Type
  Referenced from: /usr/local/lib/python3.5/site-packages/pybertini.so
  Expected in: flat namespace
 in /usr/local/lib/python3.5/site-packages/pybertini.so
ofloveandhate commented 7 years ago

it looks like Boost.Python wasn't linked to my pybertini.so:

otool -L /usr/local/lib/python3.5/site-packages/pybertini.so 
/usr/local/lib/python3.5/site-packages/pybertini.so:
    /usr/local/lib/libbertini2.0.dylib (compatibility version 1.0.0, current version 1.0.0)
    /usr/local/opt/boost/lib/libboost_serialization-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/opt/boost/lib/libboost_system.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/opt/boost/lib/libboost_filesystem.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/opt/boost/lib/libboost_chrono-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/opt/boost/lib/libboost_regex-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/opt/boost/lib/libboost_thread-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/opt/boost/lib/libboost_timer-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/opt/boost/lib/libboost_log-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/opt/boost/lib/libboost_log_setup-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/opt/mpfr/lib/libmpfr.4.dylib (compatibility version 6.0.0, current version 6.4.0)
    /usr/local/opt/gmp/lib/libgmp.10.dylib (compatibility version 14.0.0, current version 14.1.0)
    /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)
ofloveandhate commented 7 years ago

the fault is with my boost-python install. it wasn't installed with Python3 support.

brew uninstall boost-python brew install boost-python --with-python3

and now i have python3 support from Boost.Python. The evidence in in the ./configure PYTHON=python3 step, which now looks better (relevant part only):

checking consistency of all components of python development environment... yes
checking whether the Boost::Python library is available... yes
checking whether boost_python is the correct library... no
checking whether boost_python-mt is the correct library... no
checking whether boost_python-mt is the correct library... (cached) no
checking whether boost_python is the correct library... (cached) no
checking whether boost_python is the correct library... (cached) no
checking whether boost_python3-mt is the correct library... yes

before, it didn't mark any boost_python library as "the correct library", which is why I stated in my comment from 20 minutes ago that ./configure "ran kinda ok".

indeed. the output from otool now show that boost python is linked into PyBertini:

integral:~ ofloveandhate$ otool -L /usr/local/lib/python3.5/site-packages/pybertini.so 
/usr/local/lib/python3.5/site-packages/pybertini.so:
    /usr/local/opt/boost-python/lib/libboost_python3-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/lib/libbertini2.0.dylib (compatibility version 1.0.0, current version 1.0.0)
    /usr/local/opt/boost/lib/libboost_serialization-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/opt/boost/lib/libboost_system.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/opt/boost/lib/libboost_filesystem.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/opt/boost/lib/libboost_chrono-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/opt/boost/lib/libboost_regex-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/opt/boost/lib/libboost_thread-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/opt/boost/lib/libboost_timer-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/opt/boost/lib/libboost_log-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/opt/boost/lib/libboost_log_setup-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/opt/mpfr/lib/libmpfr.4.dylib (compatibility version 6.0.0, current version 6.4.0)
    /usr/local/opt/gmp/lib/libgmp.10.dylib (compatibility version 14.0.0, current version 14.1.0)
    /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.1.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)

however, I get the same errors upon trying to load PyBertini. 

ImportError: dlopen(/usr/local/lib/python3.5/site-packages/pybertini.so, 2): Symbol not found: _PyInt_Type

i will keep hunting.

ofloveandhate commented 7 years ago

upon re-compiling PyBertini after re./configureing with PYTHON=python3, and re-installing, this went away.

now, upon trying to run the tests, i am getting fails due to missing parentheses about a print thing.

integral:python ofloveandhate$ python3 test/
__init__.py  classes/     test_all.py  tracking/    
integral:python ofloveandhate$ python3 test/test_all.py 
Traceback (most recent call last):
  File "test/test_all.py", line 29, in <module>
    import classes.test_classes
  File "/Users/ofloveandhate/math_projects/bertini2/bertiniteam/b2/python/test/classes/test_classes.py", line 42
    print mods
             ^
SyntaxError: Missing parentheses in call to 'print'

however, it builds correctly for me, and i can import, so i am calling this issue solved

cthoyt commented 7 years ago

That's awesome! Thanks for following up on this. I'd suggest switching all python2 code to use from __future__ import print_function and then using python3 style printing :)