Build on Macos

[fabiencastan]

Need to test all plugins and complete the documentation.

[mfe]

WIP. Need to validate build with a CI machine

[barbak]

I have tested some compilation fixes for osx build. When doing scons check_libs=1 we have now the following output. (If you have a correct sconf file -- see tools/sconf/macos_ports.sconf for an example)

Check_libs output

Checking for pthread_attr_t attr;pthread_attr_init(&attr) in C library pthread... yes
Checking for C++ header file boost/static_assert.hpp... yes
Checking for C++ header file boost/gil/gil_all.hpp... yes
Checking for C library boost_unit_test_framework-mt... yes
Checking for C++ library boost_system-mt... yes
Checking for C++ library boost_filesystem-mt... yes
Checking for C++ library boost_regex-mt... yes
Checking for C library python... yes
Checking for C library dl... yes
Checking for C library GL... no
Checking for C++ library boost_serialization-mt... yes
Checking for C++ library boost_thread-mt... yes
Checking for C++ header file boost/atomic.hpp... yes
Checking for C++ library boost_program_options-mt... yes
Checking for C library z... yes
Checking for C++ library png... yes
Checking for C library npymath... yes
Checking for C library freetype... yes
Checking for C++ library boost_python-mt... yes
Checking for C++ library Half... yes
Checking for C++ library IlmThread... yes
Checking for Imf::InputFile("test.exr") in C++ library IlmImf... yes
Checking for C library bz2... yes
Checking for C library avutil... yes
Checking for C library lcms... yes
Checking for TIFFGetVersion() in C library tiff... yes
Checking for C library jpeg... yes
Checking for C library Xt... yes
Checking for C library m... yes
Checking for C library gomp... yes
Checking for C library ltdl... yes
Checking for C library MagickCore... yes
Checking for C library openjpeg... yes
Checking for C++ library OpenImageIO... yes
Checking for C library lcms... yes
Checking for C library raw... yes
Checking for C library turbojpeg... yes
Checking for C library GLU... yes
Checking for C library glut... yes
Checking for C library GLEW... yes
Checking for Ctl::SimdInterpreter interp; interp.setMaxInstCount(10) in C++ library IlmCtlSimd... yes
Checking for C++ library OpenColorIO... yes
Checking for C library caca... yes
Error in 'gl' library :

Configure errors... Can't start compilation!

Don't know why GL testing doesn't work.

On the compilation scons --keep-going I have the following files which don't compile:

Error list

Failed building .dist/Kawai.local/gcc-4.2/release/libraries/tuttle/src/tuttle/host/tuttle_wrap.os: Error 1
Failed building .dist/Kawai.local/gcc-4.2/release/applications/example/buffer/src/main.o: Error 1
Failed building .dist/Kawai.local/gcc-4.2/release/applications/example/canny/src/main.o: Error 1
Failed building .dist/Kawai.local/gcc-4.2/release/applications/example/io/src/main.o: Error 1
Failed building .dist/Kawai.local/gcc-4.2/release/applications/example/simpleInputBuffer/src/main.o: Error 1
Failed building .dist/Kawai.local/gcc-4.2/release/applications/example/simpleNodes/src/main.o: Error 1
Failed building .dist/Kawai.local/gcc-4.2/release/applications/sam/sam: Error 1
Failed building .dist/Kawai.local/gcc-4.2/release/applications/sam/src/sam/diff/main.o: Error 1
Failed building .dist/Kawai.local/gcc-4.2/release/applications/sam/src/sam/check/main.o: Error 1

Error details

[...]
.dist/Kawai.local/gcc-4.2/release/libraries/tuttle/src/tuttle/host/tuttle_wrap.cc:58370:
error: no matching function for call to 'tuttle::host::ComputeOptions::ComputeOptions(tuttle::host::ComputeOptions)'
libraries/tuttle/src/tuttle/host/Graph.hpp:147: error: in passing argument 1 of 'bool tuttle::host::Graph::compute(const tuttle::host::ComputeOptions&)'
.dist/Kawai.local/gcc-4.2/release/libraries/tuttle/src/tuttle/host/tuttle_wrap.cc: In function 'PyObject* _wrap_Graph_compute__SWIG_3(PyObject*, PyObject*)':
.dist/Kawai.local/gcc-4.2/release/libraries/tuttle/src/tuttle/host/tuttle_wrap.cc:58472: error: no matching function for call to 'tuttle::host::ComputeOptions::ComputeOptions(tuttle::host::ComputeOptions)'
[...]
applications/example/buffer/src/main.cpp:79: error: no matching function for call to 'tuttle::host::ComputeOptions::ComputeOptions(tuttle::host::ComputeOptions)'
libraries/tuttle/src/tuttle/host/Graph.hpp:155: error: in passing argument 3 of 'bool tuttle::host::Graph::compute(tuttle::host::memory::MemoryCache&, const tuttle::host::NodeListArg&, const tuttle::host::ComputeOptions&)'
[...]
applications/example/canny/src/main.cpp:191: error: no matching function for call to 'tuttle::host::ComputeOptions::ComputeOptions(tuttle::host::ComputeOptions)'
libraries/tuttle/src/tuttle/host/Graph.hpp:155: error: in passing argument 3 of 'bool tuttle::host::Graph::compute(tuttle::host::memory::MemoryCache&, const tuttle::host::NodeListArg&, const tuttle::host::ComputeOptions&)'
[...]
applications/example/io/src/main.cpp:89: error: no matching function for call to 'tuttle::host::ComputeOptions::ComputeOptions(tuttle::host::ComputeOptions)'
libraries/tuttle/src/tuttle/host/Graph.hpp:151: error: in passing argument 2 of 'bool tuttle::host::Graph::compute(const tuttle::host::NodeListArg&, const tuttle::host::ComputeOptions&)'
[...]
applications/example/simpleInputBuffer/src/main.cpp:67: error: no matching function for call to 'tuttle::host::ComputeOptions::ComputeOptions(tuttle::host::ComputeOptions)'
libraries/tuttle/src/tuttle/host/Graph.hpp:151: error: in passing argument 2 of 'bool tuttle::host::Graph::compute(const tuttle::host::NodeListArg&, const tuttle::host::ComputeOptions&)'
[...]
applications/example/simpleNodes/src/main.cpp:17: error: no matching function for call to 'tuttle::host::ComputeOptions::ComputeOptions(tuttle::host::ComputeOptions)'
libraries/tuttle/src/tuttle/host/Node.hpp:19: error: in passing argument 2 of 'bool tuttle::host::compute(const std::vector >&, const tuttle::host::ComputeOptions&)'
[...]
applications/sam/src/sam/diff/main.cpp:67: error: no matching function for call to 'tuttle::host::ComputeOptions::ComputeOptions(tuttle::host::ComputeOptions)'
libraries/tuttle/src/tuttle/host/Graph.hpp:151: error: in passing argument 2 of 'bool tuttle::host::Graph::compute(const tuttle::host::NodeListArg&, const tuttle::host::ComputeOptions&)'
[...]
Linking : .dist/Kawai.local/gcc-4.2/release/applications/sam/sam 
g++ -o .dist/Kawai.local/gcc-4.2/release/applications/sam/sam   .dist/Kawai.local/gcc-4.2/release/applications/sam/src/sam/sam/main.o $( -Ldist/Kawai.local/gcc-4.2/release/lib -L/opt/local/lib $) -lsequenceParser -ltuttleCommon -lboost_program_options-mt -lboost_regex-mt -lboost_filesystem-mt -lboost_system-mt -ldl -lpthread   
Undefined symbols for architecture x86_64:
  "_CFStringGetLength", referenced from:
      tuttle::common::CFStringContainer::toString(__CFString const*)in libtuttleCommon.a(macos.os)
  "_CFStringGetCharacters", referenced from:
      tuttle::common::CFStringContainer::toString(__CFString const*)in libtuttleCommon.a(macos.os)
  "_kCFAllocatorNull", referenced from:
      tuttle::common::CFStringContainer::operator __CFString const*() constin libtuttleCommon.a(macos.os)
  "_CFStringCreateWithCharactersNoCopy", referenced from:
      tuttle::common::CFStringContainer::operator __CFString const*() constin libtuttleCommon.a(macos.os)
  "_CFStringCreateWithCharacters", referenced from:
      tuttle::common::CFStringContainer::toCFStringRef(std::basic_string, std::allocator > const&)in libtuttleCommon.a(macos.os)
  "_CFBundleGetMainBundle", referenced from:
      tuttle::common::applicationFilepath(std::basic_string, std::allocator > const&, boost::filesystem::path const&)in libtuttleCommon.a(applicationPath.os)
  "_CFBundleCopyExecutableURL", referenced from:
      tuttle::common::applicationFilepath(std::basic_string, std::allocator > const&, boost::filesystem::path const&)in libtuttleCommon.a(applicationPath.os)
  "_CFURLCopyFileSystemPath", referenced from:
      tuttle::common::applicationFilepath(std::basic_string, std::allocator > const&, boost::filesystem::path const&)in libtuttleCommon.a(applicationPath.os)
  "_CFRelease", referenced from:
      tuttle::common::applicationFilepath(std::basic_string, std::allocator > const&, boost::filesystem::path const&)in libtuttleCommon.a(applicationPath.os)
      tuttle::common::CFStringContainer::~CFStringContainer()in libtuttleCommon.a(applicationPath.os)
ld: symbol(s) not found for architecture x86_64

Anyone has an idea ? Maybe a merge is missing for the compute options issue ?

[MarcAntoine-Arnaud]

In fact we need to link with the CoreFoundation (CF) Framework to have a correct build. I ask to Fabien how to do that with scons, I will test this afternoon ...

[fabiencastan]

@barbak could you send the "config.log" file? To see the error with GL.

[barbak]

@fabiencastan OSX build config.log

[barbak]

@MarcAntoine-Arnaud Cool :)

[barbak]

@fabiencastan GL check lib fails because it tries to compile with AGL/gl.h header and libOpenGL ignoring the configuration lib_gl=['GL'] in user.sconf, and AGL/gl.h is not necessarily used on osx (thanks to macports and X11) Maybe we have to discuss about Apple Frameworks handling via sconsProject, it is still a bit clumsy to use for me.

[fabiencastan]

For GL, I don't know how to do the check, maybe directly with "gl.h"... For CoreFoundation, it's just a new lib to declare in "autoconf/macos_core.py" with "LibChecker". For Apple Frameworks, sconsProject use frameworks when we declare "fwkdir_myLibrary" in "host.sconf" file. If you want to use standard directories put "True" instead of a path ;)

[barbak]

I added a corefoundation support in sconsProject. I used LibWithHeaderChecker instead of a simple LibChecker because LibChecker can't properly manage a framework (invalid num args for calling the function privateCheckFramework) Because of that I have to indicate an incdir in the sconf file to pass the check_libs test. But know it compiles better. Still have an issue with ComputeOptions and Graph.compute.

Here you can find the compilations errors http://pastebin.com/fd5bv55q

[barbak]

The error disappeared when using gcc 4.5.4 from MacPorts. Maybe a bug in the 4.2 serie ?

[cbenhagen]

i686-apple-darwin11-llvm-gcc-4.2 (GCC) 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.11.00) https://gist.github.com/cbenhagen/28d8f7485f4b1382a3d2

[fabiencastan]

@cbenhagen Are you still stuck on this error? Even after a full rebuild?

[cbenhagen]

@fabiencastan yes. --> https://gist.github.com/cbenhagen/1ebf183c427d104aa885

[fabiencastan]

I'm not sure if it's the only problem, but you need to install "numpy".

I need to declare this dependency into scons... I made a mistake by removing the fix from @barbak for that. I was thinking that it was just a python dependency and not a C++ one. But it's used in the SWIG binding.

[cbenhagen]

Numpy is installed at /usr/local/lib/python2.7/site-packages/numpy/but not found because of 25efb442b1dcd37c7fd68c9dff0ec1045452a680. If i revert this commit i still get the following errors: https://gist.github.com/cbenhagen/80edc8fb373a596fea82

[fabiencastan]

I fixed the dependency to numpy in develop https://github.com/tuttleofx/TuttleOFX/commit/c49145fb153bfc157d39345af472022956e147a1.

[fabiencastan]

Could you send me your file: .dist/Bens-MacBook-Pro.local/gcc-4.2.1/production/libraries/sequenceParser/src/sequenceParser_wrap.cc

[fabiencastan]

Could you check your SWIG version? swig -version

[cbenhagen]

sequenceParser_wrap.cc -> https://gist.github.com/cbenhagen/49ea1b6958f8d7060e99

swig -version

SWIG Version 2.0.9

Compiled with c++ [i386-apple-darwin12.2.0]

Configured options: +pcre

[fabiencastan]

Could you pull the "develop" and try again. I just removed the explicit copy constructor of "ComputeOptions".

[cbenhagen]

https://gist.github.com/cbenhagen/d9325420fcc64b2a85c1

[cbenhagen]

sequenceParser_wrap.cc --> https://gist.github.com/cbenhagen/5928b9f2ab635419e135

[fabiencastan]

It's seems that there is 3 problems:

1/ ComputeOptions copy constructor It's resolved.

2/ Py_Initialize Could you try to add "#include " in "plugins/image/generator/Text/src/TextProcess.tcc". We include "boost/python.hpp", but maybe depending on the boost version it could include "pythonrun.h" or not...

3/ Cannot convert 'PySliceObject' to 'PyObject' for argument '1' to 'int PySlice_Check(PyObject*)' No idea... The generated code for that is the same on my version, but it builds fine with gcc 4.7 and python 2.7... Which version of python are you using?

[cbenhagen]

2/ sorry for the dumb question. i have no experience with c or c++. i just add a newline at the top reading #include "pythonrun.h" right? :) with this added i do not get any further..

3/ my python version is 2.7.3

[fabiencastan]

2/ Yes And you still have the error "there are no arguments to 'Py_Initialize' that depend on a template parameter, so a declaration of 'Py_Initialize' must be available" on the "TextProcess.tcc" ?

3/ I have exactly the same version... so the only difference is the gcc version. I tried to install gcc 4.2 without success (some dependencies to libstdc++ are break...). I have no solution to propose for that point. This code is generated by SWIG and the error is on a basic function of SWIG, so I can't do anything. Could you try with a more recent version of gcc ?

[cbenhagen]

2/ yes.. still the same error

3/ maybe we could ask some SWIG guys/gals for help? seems to be a problem of some backported API from python3. fedora seems to patch its swig packages. --> https://bugzilla.redhat.com/show_bug.cgi?id=666429. i'll try to rebuild with a newer version of gcc. would i need to rebuild all dependencies with the new gcc version?

[fabiencastan]

3/ I don't know, it depends if there is an ABI change between gcc 4.2 and the version you will use.

To build with a custom gcc version: scons CC=gcc-4.7 CXX=g++-4.7

If this version is not in your path you could put an absolute path: scons CC=/path/to/gcc-4.7 CXX=/path/to/g++-4.7

You could put this into the host.sconf too. CC = "/path/to/gcc-4.7" CXX = "/path/to/g++-4.7"

[cbenhagen]

i am compiling gcc 4.7.2 right now. but from my point of view this seems more likely to be a problem of swig. what linux distribution are you running?

[fabiencastan]

I'm on Ubuntu 12.10. Some other main developers of the project use OpenSuse.

[cbenhagen]

if only i could use linux at work.. :-/ can't find any related patches in the ubuntu swig package

[fabiencastan]

My version of SWIG is 2.0.7 and the generated source code is the same on that point. So we use quite the same version of swig and the same version of python (2.7.3)... Really strange.

[fabiencastan]

Could you check that with the correct path to your python headers? grep -Hnr PySlice_Check /usr/include/python2.7/

In my python code, PySlice_Check is not a function but a define... so it seems incompatible with your gcc error message "int PySlice_Check(PyObject*)"

[cbenhagen]

/usr/local/Frameworks/Python.framework/Versions/2.7/include/python2.7/sliceobject.h:30:#define PySlice_Check(op) (Py_TYPE(op) == &PySlice_Type)

[cbenhagen]

i get the same error with gcc 4.7.2

[fabiencastan]

That's the same code... And at the end Py_TYPE itself is a "define" which do a C cast:

define Py_TYPE(ob) (((PyObject*)(ob))->ob_type)

So I don't understand how we can get this error message from that code!

[fabiencastan]

Are you sure that you use python 2.7 ?

On your build command line it's only specified: "-I/usr/local/Frameworks/Python.framework/Headers" without specific version.

[fabiencastan]

It tried with python 3.2 but it works.

[cbenhagen]

found the error :) homebrew links all its stuff from /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/ to /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/ but not the Headers direcory. It was there but empty.. need to find out why..

all i had to do is to is to specify incdir_python = '/usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Headers/' in host.sconf

[fabiencastan]

Great !

I tried with python 3.3 but it works too. Do you know what version of python was used to get this error?

[cbenhagen]

i am not sure what header files it took instead. can you propose a way to find out?

[fabiencastan]

scons pySequenceParser CCFLAGS=-H Will print all headers used. So you could grep python inside.

Remove your fix in the host.sconf before ;)

[cbenhagen]

i do not find it in there .. here is the output: https://gist.github.com/cbenhagen/d7373dcb881f24812568

[fabiencastan]

It uses "/usr/local/include/Python.h". So: grep -Hnr PY_VERSION /usr/local/include

[cbenhagen]

/usr/local/include/boost/python/converter/builtin_converters.hpp:93:#if PY_VERSION_HEX >= 0x03000000
/usr/local/include/boost/python/converter/builtin_converters.hpp:112:#if PY_VERSION_HEX >= 0x02030000
/usr/local/include/boost/python/converter/builtin_converters.hpp:125:# if defined(_MSC_VER) && defined(_WIN64) && PY_VERSION_HEX < 0x03000000
/usr/local/include/boost/python/converter/builtin_converters.hpp:155:#if PY_VERSION_HEX >= 0x03000000
/usr/local/include/boost/python/converter/pyobject_traits.hpp:37:#if PY_VERSION_HEX < 0x03000000
/usr/local/include/boost/python/detail/operator_id.hpp:50:#if PY_VERSION_HEX >= 0x03000000
/usr/local/include/boost/python/detail/operator_id.hpp:56:#if PY_VERSION_HEX >= 0x03000000
/usr/local/include/boost/python/detail/wrap_python.hpp:179:#if PY_VERSION_HEX < 0x02060000
/usr/local/include/boost/python/enum.hpp:82:#if PY_VERSION_HEX >= 0x03000000
/usr/local/include/boost/python/list.hpp:40:#if PY_VERSION_HEX >= 0x03000000
/usr/local/include/boost/python/list.hpp:120:#if PY_VERSION_HEX <= 0x03000000
/usr/local/include/boost/python/module_init.hpp:16:#  if PY_VERSION_HEX >= 0x03000000
/usr/local/include/boost/python/module_init.hpp:28:#  if PY_VERSION_HEX >= 0x03000000
/usr/local/include/boost/python/object/iterator.hpp:132:#if PY_VERSION_HEX >= 0x03000000
/usr/local/include/boost/python/operators.hpp:215:#if PY_VERSION_HEX >= 0x03000000
/usr/local/include/boost/python/operators.hpp:348:#if PY_VERSION_HEX >= 0x03000000
/usr/local/include/boost/python/ssize_t.hpp:13:#if PY_VERSION_HEX >= 0x02050000
/usr/local/include/boost/python/str.hpp:40:#if PY_VERSION_HEX < 0x03000000
/usr/local/include/boost/python/str.hpp:190:#if PY_VERSION_HEX < 0x03000000
/usr/local/include/boost/python/str.hpp:411:#if PY_VERSION_HEX >= 0x03000000
/usr/local/include/patchlevel.h:29:#define PY_VERSION       "2.7.2"
/usr/local/include/patchlevel.h:32:#define PYPY_VERSION "1.9.0"
/usr/local/include/patchlevel.h:39:   Use this for numeric comparisons, e.g. #if PY_VERSION_HEX >= ... */
/usr/local/include/patchlevel.h:40:#define PY_VERSION_HEX ((PY_MAJOR_VERSION << 24) | \
/usr/local/include/pycobject.h:42:#if (PY_VERSION_HEX >= 0x02060000 || defined(Py_BUILD_CORE))
/usr/local/include/sip.h:49:#if PY_VERSION_HEX < 0x02030000
/usr/local/include/sip.h:202:#if PY_VERSION_HEX >= 0x02050000
/usr/local/include/sip.h:602:#if PY_VERSION_HEX >= 0x02050000
/usr/local/include/sip.h:1569:#if PY_VERSION_HEX >= 0x03020000

[fabiencastan]

Curious to see the result of: grep -Hnr PySlice_Check /usr/local/include

[fabiencastan]

Python 2.7.2

[cbenhagen]

/usr/local/include/boost/python/suite/indexing/indexing_suite.hpp:209:            if (PySlice_Check(i))
/usr/local/include/boost/python/suite/indexing/indexing_suite.hpp:219:            if (PySlice_Check(i))
/usr/local/include/boost/python/suite/indexing/indexing_suite.hpp:258:            if (PySlice_Check(i))
/usr/local/include/pypy_decl.h:356:PyAPI_FUNC(int) PySlice_Check(PyObject *arg0);
/usr/local/include/pypy_decl.h:357:PyAPI_FUNC(int) PySlice_CheckExact(PyObject *arg0);

[fabiencastan]

Ok, so PySlice_Check was a function in Python 2.7.2 ! Thanks

[cbenhagen]

i still dont really understand how that happened.. but as long as you do im fine with it :)

btw: /usr/local/include/Python.h is a symlink to /usr/local/Cellar/pypy/1.9/include/Python.h

so for now the last build error is in TextPlugin: https://gist.github.com/cbenhagen/721cd54c9575fd0fdef6

[fabiencastan]

grep -Hnr Py_Initialize /usr/local/Cellar/python/2.7.3/Frameworks/Python.framework/Headers/