Python Wrapping Failing

moorepants commented 10 years ago

I'm trying to build on Ubuntu 13.10 64 bit and having issues with the Python wrappings being built properly.

moorepants@moorepants-2170p:~$ git clone git@github.com:Biomechanical-ToolKit/BTKCore.git ~/src/BTKCore
moorepants@moorepants-2170p:~$ git clone git@github.com:Biomechanical-ToolKit/BTKData.git ~/Data/BTKData
moorepants@moorepants-2170p:~$ cd ~/src/BTKCore
# last working commit on the CI linux machine on the master branch
moorepants@moorepants-2170p:BTKCore(master)$ git checkout 8971e194fefdc14
moorepants@moorepants-2170p:BTKCore((8971e19...))$ git checkout -b new-numpy-i
moorepants@moorepants-2170p:BTKCore(new-numpy-i)$ wget https://raw.github.com/numpy/numpy/master/doc/swig/numpy.i -O Utilities/SWIG/numpy.i
moorepants@moorepants-2170p:BTKCore(new-numpy-i)$ mkdir build
moorepants@moorepants-2170p:BTKCore(new-numpy-i)$ cd build
moorepants@moorepants-2170p:build(new-numpy-i)$ cmake \
>    -DCMAKE_BUILD_TYPE:CHAR=Release \
>    -DBUILD_SHARED_LIBS:BOOL=1 \
>    -DBTK_WRAP_PYTHON:BOOL=1 \
>    -DBTK_WRAP_OCTAVE:BOOL=1 \
>    -DBUILD_TESTING:BOOL=1 \
>    -DBTK_TESTING_DATA_PATH:CHAR=~/Data/BTKData \
>    -DBTK_EXTRA_COMPILER_WARNINGS:BOOL=1 \
>    -DBUILD_DOCUMENTATION:BOOL=1 \
>    -DBUILD_DOCUMENTATION_API:BOOL=1 \
>    -DBUILD_DOCUMENTATION_API_UNSELECTED_MODULES:BOOL=1 \
>    -DBUILD_DOCUMENTATION_MOKKA:BOOL=1 \
>    -DBUILD_DOCUMENTATION_README:BOOL=1 \
>    -DBUILD_EXAMPLES:BOOL=1 \
>    -G "Unix Makefiles" ..
-- The C compiler identification is GNU 4.8.1
-- The CXX compiler identification is GNU 4.8.1
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Looking for sys/mman.h
-- Looking for sys/mman.h - found
-- Looking for C++ include memory
-- Looking for C++ include memory - found
-- Looking for C++ include tr1/memory
-- Looking for C++ include tr1/memory - found
-- Looking for boost/memory.hpp
-- Looking for boost/memory.hpp - not found
-- Looking for boost/tr1/memory.hpp
-- Looking for boost/tr1/memory.hpp - not found
-- Looking for include file pthread.h
-- Looking for include file pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Checking for builtin __sync_add_and_fetch
-- Checking for builtin __sync_add_and_fetch -- success
-- Performing Test HAVE_GCC_ERROR_RETURN_TYPE
-- Performing Test HAVE_GCC_ERROR_RETURN_TYPE - Success
-- Found SWIG: /usr/bin/swig2.0 (found suitable version "2.0.10", minimum required is "2.0") 
-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.3m.so (found version "3.3.2+") 
-- Found PythonInterp: /usr/bin/python (found version "2.7.5") 
-- Found NumPy: 1.7.1  
-- Found Octave: /usr/bin/octave  
-- Found Doxygen: /usr/bin/doxygen (found version "1.8.4") 
-- Configuring done
-- Generating done
-- Build files have been written to: /home/moorepants/src/BTKCore/build

Notice that cmake finds the python 3 shared library but the python 2.7.5 interpreter. The build then fails:

moorepants@moorepants-2170p:build(new-numpy-i)$ make -j4
.
.
.
[100%] Building CXX object Wrapping/Python/CMakeFiles/_btk.dir/btkSwigPYTHON_wrap.cxx.o
/home/moorepants/src/BTKCore/build/Wrapping/Python/btkSwigPYTHON_wrap.cxx: In function ‘void CopyFromEigenToNumPyMatrix(PyObject*, Eigen::MatrixBase<Derived>*)’:
/home/moorepants/src/BTKCore/build/Wrapping/Python/btkSwigPYTHON_wrap.cxx:4378:93: error: cannot convert ‘PyObject* {aka _object*}’ to ‘PyArrayObject* {aka tagPyArrayObject*}’ for argument ‘1’ to ‘void* PyArray_DATA(PyArrayObject*)’
     typename Derived::Scalar* data = static_cast<typename Derived::Scalar*>(PyArray_DATA(out));
                                                                                             ^
/home/moorepants/src/BTKCore/build/Wrapping/Python/btkSwigPYTHON_wrap.cxx: In function ‘void ConvertFromEigenToNumPyMatrix(PyObject**, Eigen::MatrixBase<Derived>*)’:
/home/moorepants/src/BTKCore/build/Wrapping/Python/btkSwigPYTHON_wrap.cxx:4392:94: error: cannot convert ‘PyObject* {aka _object*}’ to ‘PyArrayObject* {aka tagPyArrayObject*}’ for argument ‘1’ to ‘void* PyArray_DATA(PyArrayObject*)’
     typename Derived::Scalar* data = static_cast<typename Derived::Scalar*>(PyArray_DATA(*out));
                                                                                              ^
/home/moorepants/src/BTKCore/build/Wrapping/Python/btkSwigPYTHON_wrap.cxx: In function ‘int NumPyType() [with T = double]’:
/home/moorepants/src/BTKCore/build/Wrapping/Python/btkSwigPYTHON_wrap.cxx:4398:46: error: ‘PyArray_DOUBLE’ was not declared in this scope
   template<> int NumPyType<double>() {return PyArray_DOUBLE;};
                                              ^
make[2]: *** [Wrapping/Python/CMakeFiles/_btk.dir/btkSwigPYTHON_wrap.cxx.o] Error 1
make[1]: *** [Wrapping/Python/CMakeFiles/_btk.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
Linking CXX executable ../../bin/TDD
[100%] Built target TDD
make: *** [all] Error 2

moorepants commented 10 years ago

Now if I explicitly point cmake to the python 2.7 shared library and python 2.7 header files:

moorepants@moorepants-2170p:build(new-numpy-i)$ cmake \
>    -DCMAKE_BUILD_TYPE:CHAR=Release \
>    -DBUILD_SHARED_LIBS:BOOL=1 \
>    -DBTK_WRAP_PYTHON:BOOL=1 \
>    -DBTK_WRAP_OCTAVE:BOOL=1 \
>    -DBUILD_TESTING:BOOL=1 \
>    -DBTK_TESTING_DATA_PATH:CHAR=~/Data/BTKData \
>    -DBTK_EXTRA_COMPILER_WARNINGS:BOOL=1 \
>    -DBUILD_DOCUMENTATION:BOOL=1 \
>    -DBUILD_DOCUMENTATION_API:BOOL=1 \
>    -DBUILD_DOCUMENTATION_API_UNSELECTED_MODULES:BOOL=1 \
>    -DBUILD_DOCUMENTATION_README:BOOL=1 \
>    -DBUILD_EXAMPLES:BOOL=1 \
>    -DPYTHON_LIBRARY:CHAR=/usr/lib/x86_64-linux-gnu/libpython2.7.so \
>    -DPYTHON_INCLUDE_DIR:CHAR=/usr/include/python2.7 \
>    -G "Unix Makefiles" ..
-- The C compiler identification is GNU 4.8.1
-- The CXX compiler identification is GNU 4.8.1
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Looking for sys/mman.h
-- Looking for sys/mman.h - found
-- Looking for C++ include memory
-- Looking for C++ include memory - found
-- Looking for C++ include tr1/memory
-- Looking for C++ include tr1/memory - found
-- Looking for boost/memory.hpp
-- Looking for boost/memory.hpp - not found
-- Looking for boost/tr1/memory.hpp
-- Looking for boost/tr1/memory.hpp - not found
-- Looking for include file pthread.h
-- Looking for include file pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Checking for builtin __sync_add_and_fetch
-- Checking for builtin __sync_add_and_fetch -- success
-- Performing Test HAVE_GCC_ERROR_RETURN_TYPE
-- Performing Test HAVE_GCC_ERROR_RETURN_TYPE - Success
-- Found SWIG: /usr/bin/swig2.0 (found suitable version "2.0.10", minimum required is "2.0") 
-- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython2.7.so (found version "2.7.5+") 
-- Found PythonInterp: /usr/bin/python (found version "2.7.5") 
-- Found NumPy: 1.7.1  
-- Found Octave: /usr/bin/octave  
-- Found Doxygen: /usr/bin/doxygen (found version "1.8.4") 
-- Configuring done
-- Generating done
-- Build files have been written to: /home/moorepants/src/BTKCore/build

I get the same error as before, so maybe the new numpy.i file isn't working.

moorepants@moorepants-2170p:build(new-numpy-i)$ make -j4
.
.
.
[100%] [100%] Building CXX object Wrapping/Python/CMakeFiles/_btk.dir/btkSwigPYTHON_wrap.cxx.o
Building CXX object Testing/C++/CMakeFiles/TDD.dir/_TDDIO.cpp.o
/home/moorepants/src/BTKCore/build/Wrapping/Python/btkSwigPYTHON_wrap.cxx: In function ‘void CopyFromEigenToNumPyMatrix(PyObject*, Eigen::MatrixBase<Derived>*)’:
/home/moorepants/src/BTKCore/build/Wrapping/Python/btkSwigPYTHON_wrap.cxx:4378:93: error: cannot convert ‘PyObject* {aka _object*}’ to ‘PyArrayObject* {aka tagPyArrayObject*}’ for argument ‘1’ to ‘void* PyArray_DATA(PyArrayObject*)’
     typename Derived::Scalar* data = static_cast<typename Derived::Scalar*>(PyArray_DATA(out));
                                                                                             ^
/home/moorepants/src/BTKCore/build/Wrapping/Python/btkSwigPYTHON_wrap.cxx: In function ‘void ConvertFromEigenToNumPyMatrix(PyObject**, Eigen::MatrixBase<Derived>*)’:
/home/moorepants/src/BTKCore/build/Wrapping/Python/btkSwigPYTHON_wrap.cxx:4392:94: error: cannot convert ‘PyObject* {aka _object*}’ to ‘PyArrayObject* {aka tagPyArrayObject*}’ for argument ‘1’ to ‘void* PyArray_DATA(PyArrayObject*)’
     typename Derived::Scalar* data = static_cast<typename Derived::Scalar*>(PyArray_DATA(*out));
                                                                                              ^
/home/moorepants/src/BTKCore/build/Wrapping/Python/btkSwigPYTHON_wrap.cxx: In function ‘int NumPyType() [with T = double]’:
/home/moorepants/src/BTKCore/build/Wrapping/Python/btkSwigPYTHON_wrap.cxx:4398:46: error: ‘PyArray_DOUBLE’ was not declared in this scope
   template<> int NumPyType<double>() {return PyArray_DOUBLE;};
                                              ^
make[2]: *** [Wrapping/Python/CMakeFiles/_btk.dir/btkSwigPYTHON_wrap.cxx.o] Error 1
make[1]: *** [Wrapping/Python/CMakeFiles/_btk.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
Linking CXX executable ../../bin/TDD
[100%] Built target TDD
make: *** [all] Error 2

Alzathar commented 10 years ago

The problem is no more related to the selection of the Python library but something wrong in the file Utilities/SWIG/eigen.i in the sources of BTKCore.

Alzathar commented 10 years ago

By the way, the use of the new numpy.i file removed your initial error which was:

moorepants@moorepants-2170p:build((8971e19...))$ make
[  7%] Built target BTKCommon
[ 12%] Built target BTKBasicFilters
[ 52%] Built target BTKIO
[ 52%] Swig source
[ 53%] Building CXX object Wrapping/Python/CMakeFiles/_btk.dir/btkSwigPYTHON_wrap.cxx.o
In file included from /usr/include/python3.3m/numpy/ndarraytypes.h:1728:0,
                 from /usr/include/python3.3m/numpy/ndarrayobject.h:17,
                 from /usr/include/python3.3m/numpy/arrayobject.h:15,
                 from /home/moorepants/src/BTKCore/build/Wrapping/Python/btkSwigPYTHON_wrap.cxx:3601:
/usr/include/python3.3m/numpy/npy_deprecated_api.h:11:2: warning: #warning "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION"
  ^
/home/moorepants/src/BTKCore/build/Wrapping/Python/btkSwigPYTHON_wrap.cxx: In function ‘const char* pytype_string(PyObject*)’:
/home/moorepants/src/BTKCore/build/Wrapping/Python/btkSwigPYTHON_wrap.cxx:3876:32: error: ‘PyFile_Check’ was not declared in this scope
     if (PyFile_Check(    py_obj)) return "file"        ;
                                ^
/home/moorepants/src/BTKCore/build/Wrapping/Python/btkSwigPYTHON_wrap.cxx:3878:32: error: ‘PyInstance_Check’ was not declared in this scope
     if (PyInstance_Check(py_obj)) return "instance"    ;
                                ^
make[2]: *** [Wrapping/Python/CMakeFiles/_btk.dir/btkSwigPYTHON_wrap.cxx.o] Error 1
make[1]: *** [Wrapping/Python/CMakeFiles/_btk.dir/all] Error 2
make: *** [all] Error 2

I need to update this file for Python 3.

moorepants commented 10 years ago

Ok. Just for completeness I ran with the numpy.i in 8971e19 and get this error, the original one, as you said.::

moorepants@moorepants-2170p:build((8971e19...))$ make
.
.
.
[100%] Building CXX object Testing/C++/CMakeFiles/TDD.dir/_TDDIO.cpp.o
In file included from /usr/include/python2.7/numpy/ndarraytypes.h:1728:0,
                 from /usr/include/python2.7/numpy/ndarrayobject.h:17,
                 from /usr/include/python2.7/numpy/arrayobject.h:15,
                 from /home/moorepants/src/BTKCore/build/Wrapping/Python/btkSwigPYTHON_wrap.cxx:3601:
/usr/include/python2.7/numpy/npy_deprecated_api.h:11:2: warning: #warning "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
 #warning "Using deprecated NumPy API, disable it by #defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION"
  ^
Linking CXX shared module ../../bin/_btk.so
[100%] Built target _btk
Linking CXX executable ../../bin/TDD
[100%] Built target TDD

Alzathar commented 10 years ago

I do not have this warning with Clang under MacOS X (and maybe also because I use SWIG 2.0.11). Can you tell me what contain the line /home/moorepants/src/BTKCore/build/Wrapping/Python/btkSwigPYTHON_wrap.cxx:3601 ?

moorepants commented 10 years ago

Line 3601 is:

#include <numpy/arrayobject.h>

moorepants commented 10 years ago

SWIG 2.0.10 is the current SWIG in the Ubuntu Saucy repos. I could manually install a new version, but aren't you using an even lower version of SWIG on your Linux CI box, since it is Ubuntu 12.04?

Alzathar commented 10 years ago

Yes, it is the case, The linux CI box uses SWIG 2.0.4.

I would say the current problem is not related to SWIG but a modification in the C API of Numpy because you have these errors:

/home/moorepants/src/BTKCore/build/Wrapping/Python/btkSwigPYTHON_wrap.cxx: In function ‘void CopyFromEigenToNumPyMatrix(PyObject*, Eigen::MatrixBase<Derived>*)’:
/home/moorepants/src/BTKCore/build/Wrapping/Python/btkSwigPYTHON_wrap.cxx:4378:93: error: cannot convert ‘PyObject* {aka _object*}’ to ‘PyArrayObject* {aka tagPyArrayObject*}’ for argument ‘1’ to ‘void* PyArray_DATA(PyArrayObject*)’
     typename Derived::Scalar* data = static_cast<typename Derived::Scalar*>(PyArray_DATA(out));
                                                                                             ^
/home/moorepants/src/BTKCore/build/Wrapping/Python/btkSwigPYTHON_wrap.cxx: In function ‘void ConvertFromEigenToNumPyMatrix(PyObject**, Eigen::MatrixBase<Derived>*)’:
/home/moorepants/src/BTKCore/build/Wrapping/Python/btkSwigPYTHON_wrap.cxx:4392:94: error: cannot convert ‘PyObject* {aka _object*}’ to ‘PyArrayObject* {aka tagPyArrayObject*}’ for argument ‘1’ to ‘void* PyArray_DATA(PyArrayObject*)’
     typename Derived::Scalar* data = static_cast<typename Derived::Scalar*>(PyArray_DATA(*out));
                                                                                              ^
/home/moorepants/src/BTKCore/build/Wrapping/Python/btkSwigPYTHON_wrap.cxx: In function ‘int NumPyType() [with T = double]’:
/home/moorepants/src/BTKCore/build/Wrapping/Python/btkSwigPYTHON_wrap.cxx:4398:46: error: ‘PyArray_DOUBLE’ was not declared in this scope
   template<> int NumPyType<double>() {return PyArray_DOUBLE;};

This code comes from the SWIG fragments in the file Utilities/SWIG/eigen.i. By modifying the way I extract the data from the PyObject pointer, it should fix the problem.

Alzathar commented 10 years ago

The commit 8664db32a136a6fcd7e002c5744f0757a01a1110 should fix the build of the Python Wrapping under Ubuntu 13.10. Can you try it?

moorepants commented 10 years ago

Ok, it now compiles and the tests pass. I simply checked out 8664db3 (I am not using the new numpy.i file) and ran my previous cmake commands. All is good. Thanks so much. I'm looking forward to trying out the software.

Biomechanical-ToolKit / BTKCore

Python Wrapping Failing #1