Compilation on Mac - Notes for futur users

[stephane-lb]

Hi,

For futur users : 1- launch these commands to go from python 2 to 3

2to3 -w ./build/SConscript.install 
2to3 -w ./build/SConscript.configure   

2- Use this file config-macos10.12.6-clang-x86_64.py

BUILDDIR       = '#build/release'
DISTDIR        = '#Mitsuba.app'
CXX            = 'clang++'
CC             = 'clang'
CCFLAGS        = ['-mmacosx-version-min=10.12', '-march=native', '-funsafe-math-optimizations', '-fno-math-errno', '-isysroot', '/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk', '-O3', '-Wall', '-Wno-deprecated-declarations', '-g', '-DMTS_DEBUG', '-DSINGLE_PRECISION', '-DSPECTRUM_SAMPLES=3', '-DMTS_SSE', '-DMTS_HAS_COHERENT_RT', '-fvisibility=hidden', '-ftemplate-depth=512', '-stdlib=libc++']
LINKFLAGS      = ['-framework', 'OpenGL', '-framework', 'Cocoa', '-mmacosx-version-min=10.12', '-Wl,-syslibroot,/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.13.sdk', '-Wl,-headerpad,128', '-stdlib=libc++']
CXXFLAGS       = ['-std=c++11']
BASEINCLUDE    = ['#include', '#dependencies/include']
BASELIBDIR     = ['#dependencies/lib']
BASELIB        = ['m', 'pthread', 'Half']
OEXRINCLUDE    = ['#dependencies/include/OpenEXR']
OEXRLIB        = ['IlmImf', 'Imath', 'Iex', 'z']
PNGLIB         = ['png16']
PNGINCLUDE     = ['#dependencies/include/libpng']
JPEGLIB        = ['jpeg']
JPEGINCLUDE    = ['#dependencies/include/libjpeg']
XERCESLIB      = ['xerces-c']
GLLIB          = ['GLEWmx', 'objc']
GLFLAGS        = ['-DGLEW_MX']
BOOSTINCLUDE   = ['#dependencies']
BOOSTLIB       = ['boost_filesystem', 'boost_system', 'boost_thread']
PYTHON27INCLUDE= ['/System/Library/Frameworks/Python.framework/Versions/2.7/Headers']
PYTHON27LIBDIR = ['/System/Library/Frameworks/Python.framework/Versions/2.7/lib']
PYTHON27LIB    = ['boost_python27', 'boost_system']
PYTHON35INCLUDE= ['#dependencies/include/python3.5']
PYTHON35LIB    = ['boost_python36', 'boost_system']
PYTHON36INCLUDE= ['#dependencies/include/python3.4']
PYTHON36LIB    = ['boost_python36', 'boost_system']
# COLLADAINCLUDE = ['#dependencies/include/collada-dom', '#dependencies/include/collada-dom/1.4']
# COLLADALIB     = ['collada14dom24']
QTDIR          = '#dependencies'
FFTWLIB        = ['fftw3']

Best Regards,

stephane-lb

( Note : this is before updating to High Sierra )

[amyspark]

1- launch these commands to go from python 2 to 3

2to3 -w ./build/SConscript.install
2to3 -w ./build/SConscript.configure

Are you using SCons 3?

[BorisKourt]

Applying these changes fixed compilation for me. I have SCons 3. It seems that there is no ply support though?

I also needed to install libpng via brew to get the program to run.

[amyspark]

I also needed to install libpng via brew to get the program to run.

I've just checked that; it isn't necessary to brew libpng if one uses the precompiled dependencies.

As for updating SCons: SCons 3 is only available by default in MacOS via Homebrew, and the newest one Ubuntu ships is 2.5, which uses Python 2. Still, I will add it to the manual (#37) for future reference.

[BorisKourt]

I've been using the macos percompiled deps from here https://github.com/mitsuba-renderer/dependencies_macos I might have misspoken and it could be libjpeg though. I do know I needed to install one or the other before running. Unable to double check at the moment.

But will take a look again as soon as I can.

[stephane-lb]

@amyspark Yes using Scons 3.0.

About libpng etc... For what I just checked using otool -L it doesn't seems to be linked to local library. All the libraries are using rpath.

[hyperknot]

Can anyone from this thread have a look at why is it not detecting Qt 5 recently? https://github.com/mitsuba-renderer/mitsuba/issues/53

[andrewvarga]

Could anyone make the compile work on the Mac High Sierra? To me it looks like libpng had to be installed manually with brew, but mtsgui is still not opening, probably a problem with qt..it used to work before, with an earlier osx and an earlier git version.

[merlinND]

Hello @andrewvarga,

I've been able to compile Mitsuba on macOS 10.13 (High Sierra) just now by following these steps:

git clone git@github.com:mitsuba-renderer/mitsuba.git
cd mitsuba
git clone git@github.com:mitsuba-renderer/dependencies_macos.git dependencies
cp build/config-macos10.12-clang-x86_64.py config.py
vim config.py
# Replace 10.12 by 10.13 (four occurrences) and save.
scons

After a while, everything compiles without error. The package Mitsuba.app is created.

I'm using Scons 3 from brew:

$ scons --version
SCons by Steven Knight et al.:
    script: v3.0.1.74b2c53bc42290e911b334a6b44f187da698a668, 2017/11/14 13:16:53, by bdbaddog on hpmicrodog
    engine: v3.0.1.74b2c53bc42290e911b334a6b44f187da698a668, 2017/11/14 13:16:53, by bdbaddog on hpmicrodog
    engine path: ['/usr/local/Cellar/scons/3.0.1/libexec/scons-local/SCons']
Copyright (c) 2001 - 2017 The SCons Foundation

[andrewvarga]

Hi @merlinND , thanks for the reply! That's exactly what I did, with no success unfortunately. For me there were 4 occurrences of 10.12, replaced them to 10.13 (the sdk exists there at the location I verified it), the build completes successfully but the resulting Mitsuba.app wouldn't run, it looks to be corrupt (You can't open the application "Mitsuba.app" because it may be damaged or incomplete.).

I was using scons 2.5.1 before, now I upgraded to 3.0.1 and made a rebuild, I get the same result.

It works only with the old repository (https://www.mitsuba-renderer.org/hg/mitsuba), there I get a correct Mitsuba.app.

Is there a way to see why the .app file is incorrect, what it is missing to run? Could it be related to PNGLIB as I see that changed to png16 and others mentioned they had to install that manually also..?

[merlinND]

Running the application from the terminal may give you more information:

cd mitsuba
./Mitsuba.app/Contents/MacOS/mtsgui

I'm not sure whether that's the version being used, but I do have libpng installed from brew:

$ brew info libpng
libpng: stable 1.6.34 (bottled), HEAD
Library for manipulating PNG images
http://www.libpng.org/pub/png/libpng.html
/usr/local/Cellar/libpng/1.6.34 (26 files, 1.2MB) *
  Poured from bottle on 2017-11-30 at 18:22:25
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/libpng.rb

And here's the output otool -L to check which libraries are being loaded from where:

otool -L ./Mitsuba.app/Contents/MacOS/mtsgui
./Mitsuba.app/Contents/MacOS/mtsgui:
    /System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL (compatibility version 1.0.0, current version 1.0.0)
    /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 22.0.0)
    @rpath/QtGui (compatibility version 5.9.0, current version 5.9.1)
    @rpath/QtWidgets (compatibility version 5.9.0, current version 5.9.1)
    @rpath/QtCore (compatibility version 5.9.0, current version 5.9.1)
    @rpath/QtOpenGL (compatibility version 5.9.0, current version 5.9.1)
    @rpath/QtXml (compatibility version 5.9.0, current version 5.9.1)
    @rpath/QtXmlPatterns (compatibility version 5.9.0, current version 5.9.1)
    @rpath/QtNetwork (compatibility version 5.9.0, current version 5.9.1)
    @loader_path/../Frameworks/BWToolkitFramework.framework/Versions/A/BWToolkitFramework (compatibility version 1.0.0, current version 0.0.0)
    /System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 58286.51.6)
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1452.23.0)
    @rpath/libGLEWmx.dylib (compatibility version 1.13.0, current version 1.13.0)
    /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
    @rpath/libxerces-c.dylib (compatibility version 0.0.0, current version 0.0.0)
    @rpath/libboost_filesystem.dylib (compatibility version 0.0.0, current version 0.0.0)
    @rpath/libboost_system.dylib (compatibility version 0.0.0, current version 0.0.0)
    @rpath/libboost_thread.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.50.4)
    @rpath/libHalf.dylib (compatibility version 0.0.0, current version 0.0.0)
    @rpath/libmitsuba-core.dylib (compatibility version 0.0.0, current version 0.0.0)
    @rpath/libmitsuba-render.dylib (compatibility version 0.0.0, current version 0.0.0)
    @rpath/libmitsuba-hw.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 400.9.0)
    /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit (compatibility version 45.0.0, current version 1561.40.112)
    /System/Library/Frameworks/CoreGraphics.framework/Versions/A/CoreGraphics (compatibility version 64.0.0, current version 1161.10.0)
    /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1452.23.0)

[andrewvarga]

Actually mtsgui is missing, so that is the problem (mitsuba, mtssrc and mtsutil are there in the MacOS folder).

Could that be that the new version uses a different Qt that I should install manually? Shouldn't the build process stop if it cannot build the gui? I'm trying to find a way to see the error which causes the gui to fail to be built.

[andrewvarga]

Yep, it's not building the GUI because it cannot detect Qt for some reason: "Unable to detect a Qt installation -- not building the GUI!"

I see the same in the old and the new config file in regards to Qt: QTDIR = '#dependencies'

How do I know why the new one can't detect Qt to build the GUI?

[andrewvarga]

Yeah, it's not picking up Qt for some reason. I think it's the same problem as: https://github.com/mitsuba-renderer/mitsuba/issues/53

I even installed Qt manually (5.9.1), I have the command line tool set in Xcode like this:

This is the console after building, the only other issue I see is Python 2.7 missing, not sure if that could have anything to do with it.

Unable to detect a Qt installation -- not building the GUI!
Checking for clang++ ...(cached) yes
Checking for C header file png.h... (cached) yes
Checking for C header file jpeglib.h... (cached) yes
Checking for C++ header file ImfRgba.h... (cached) yes
Checking for C++ header file xercesc/dom/DOMLSParser.hpp... (cached) yes
Checking for C++ header file dae.h... (cached) no
COLLADA DOM is missing: not building the COLLADA importer
Checking for C++ header file pyconfig.h... (cached) no
Python 2.7 is missing: not building wrappers
Checking for C++ header file pyconfig.h... (cached) yes
Checking for C++ header file pyconfig.h... (cached) yes
Checking for C++ header file boost/version.hpp... (cached) yes
Checking for C++ header file Eigen/Core... (cached) yes
Checking for C++ header file fftw3.h... (cached) yes
Checking for C header file OpenGL/gl.h... (cached) yes
Checking for C header file OpenGL/glu.h... (cached) yes
Checking for C header file OpenGL/glext.h... (cached) yes
Checking for C header file OpenGL/glew.h... (cached) yes
Checking for Mitsuba version .. 0.6.0
scons: done reading SConscript files.
scons: Building targets ...
scons: `.' is up to date.
scons: done building targets.

[merlinND]

There might be a conflict in SCons between the version installed manually and the version provided in the dependencies directory (although I don't understand how or why). Here's the location where you'll probably get the most information:

https://github.com/mitsuba-renderer/mitsuba/blob/1fd0f671dfcb77f813c0d6a36f2aa4e480b5ca8e/build/SConscript.configure#L99-L102

I would start by printing the exception to see if it's looking in the right places, and trying to understand why this check fails.

[andrewvarga]

That's a good tip, getting closer. It seems it doesn't find the lupdate command:

Qt5 command 'lupdate' not found. Tried: 
/Users/andrew/projects/mts/mitsuba2018/dependencies/bin/lupdate-qt5, 
/Users/andrew/projects/mts/mitsuba2018/dependencies/bin/lupdate-qt5.exe, 
/Users/andrew/projects/mts/mitsuba2018/dependencies/bin/lupdate5, 
/Users/andrew/projects/mts/mitsuba2018/dependencies/bin/lupdate5.exe, 
/Users/andrew/projects/mts/mitsuba2018/dependencies/bin/lupdate, 
/Users/andrew/projects/mts/mitsuba2018/dependencies/bin/lupdate.exe

dependencies/bin only contains these executables:

moc
qmake
rcc
uic

I don't think it's a conflict with the local Qt install because I got the same error before installing Qt manually (only reason I installed was that the compile didn't work).

[merlinND]

That's a good find! Indeed, both lupdate and lrelease are missing from the bin directory provided in dependencies, but SCons is trying to locate them. On my system, they were being picked up from the path.

As a quick workaround, could you try disabling the Scons code that locates these commands and try again? We'll see if they are really needed.

diff --git a/data/scons/qt5.py b/data/scons/qt5.py
index 3c5b85dc..587c7af3 100644
--- a/data/scons/qt5.py
+++ b/data/scons/qt5.py
@@ -255,8 +255,8 @@ def generate(env):
                 QT5_MOC = locateQt5Command(env,'moc', env['QTDIR']),
                 QT5_UIC = locateQt5Command(env,'uic', env['QTDIR']),
                 QT5_RCC = locateQt5Command(env,'rcc', env['QTDIR']),
-                QT5_LUPDATE = locateQt5Command(env,'lupdate', env['QTDIR']),
-                QT5_LRELEASE = locateQt5Command(env,'lrelease', env['QTDIR']),
+                # QT5_LUPDATE = locateQt5Command(env,'lupdate', env['QTDIR']),
+                # QT5_LRELEASE = locateQt5Command(env,'lrelease', env['QTDIR']),

                 QT5_AUTOSCAN = 1, # Should the qt tool try to figure out, which sources are to be moc'ed?

@@ -288,24 +288,24 @@ def generate(env):
                 QT5_MOCFROMCXXCOM = [
                         '$QT5_MOC $QT5_MOCFROMCXXFLAGS $QT5_MOCINCFLAGS -o $TARGET $SOURCE',
                         Action(checkMocIncluded,None)],
-                QT5_LUPDATECOM = '"$QT5_LUPDATE" $SOURCE -ts $TARGET',
-                QT5_LRELEASECOM = '"$QT5_LRELEASE" $SOURCE',
+                # QT5_LUPDATECOM = '"$QT5_LUPDATE" $SOURCE -ts $TARGET',
+                # QT5_LRELEASECOM = '"$QT5_LRELEASE" $SOURCE',
                 QT5_RCCCOM = '"$QT5_RCC" $QT5_QRCFLAGS $SOURCE -o $TARGET',
                 )

         # Translation builder
-        tsbuilder = Builder(
-                action = SCons.Action.Action('$QT5_LUPDATECOM'), #,'$QT5_LUPDATECOMSTR'),
-                multi=1
-                )
-        env.Append( BUILDERS = { 'Ts': tsbuilder } )
-        qmbuilder = Builder(
-                action = SCons.Action.Action('$QT5_LRELEASECOM'),# , '$QT5_LRELEASECOMSTR'),
-                src_suffix = '.ts',
-                suffix = '.qm',
-                single_source = True
-                )
-        env.Append( BUILDERS = { 'Qm': qmbuilder } )
+        # tsbuilder = Builder(
+        #         action = SCons.Action.Action('$QT5_LUPDATECOM'), #,'$QT5_LUPDATECOMSTR'),
+        #         multi=1
+        #         )
+        # env.Append( BUILDERS = { 'Ts': tsbuilder } )
+        # qmbuilder = Builder(
+        #         action = SCons.Action.Action('$QT5_LRELEASECOM'),# , '$QT5_LRELEASECOMSTR'),
+        #         src_suffix = '.ts',
+        #         suffix = '.qm',
+        #         single_source = True
+        #         )
+        # env.Append( BUILDERS = { 'Qm': qmbuilder } )

         # Resource builder
         def scanResources(node, env, path, arg):

[andrewvarga]

Yes, this does help, the Mitsuba.app is created successfully, thank you!