Python not working in Mac Nightly build from 2017-03-21

[cquammen]

I downloaded Mac Nightly build from http://tomviz.org/files/tomviz_nightly_bin/tomviz-latest-Darwin-64bit.dmg. When trying to execute any Python operator or opening the Python console, the application crashes.

Relevant stack trace portion here:

3   libpython3.6m.dylib             0x000000011219e507 Py_FatalError + 439
4   libpython3.6m.dylib             0x000000011219e2f8 _Py_InitializeEx_Private + 1336
5   libvtkPythonInterpreter.1.dylib 0x0000000111b00df8 vtkPythonInterpreter::Initialize(int) + 24
6   libvtkpqPython.1.dylib          0x000000010c4e7d0c pqPythonManager::pythonShellDialog() + 92
7   libvtkpqApplicationComponents.1.dylib   0x000000010bf6d5bf pqPythonShellReaction::showPythonShell() + 47

[cjh1]

Interesting, I tested this on bigmac and it seems to run OK.

[cjh1]

Which as @mathturtle points out is not an ideal test as its the machine that package was built on.

[cquammen]

Indeed.

Here's the output of otool -L

terminus:MacOS cory.quammen$ otool -L tomviz 
tomviz:
    @executable_path/../Libraries/libvtkpqApplicationComponents.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkpqComponents.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkpqPython.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkpqCore.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkpqWidgets.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libQtTesting.dylib (compatibility version 0.0.0, current version 0.0.0)
    @executable_path/../Libraries/libtomvizExtensions.dylib (compatibility version 0.0.0, current version 0.0.0)
    @executable_path/../Libraries/libvtkPVAnimation.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkPVServerManagerDefault.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkPVServerManagerRendering.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkPVServerImplementationRendering.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkPVClientServerCoreRendering.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkDomainsChemistry.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkRenderingLabel.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkViewsContext2D.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkViewsCore.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkPVVTKExtensionsDefault.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkPVVTKExtensionsRendering.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkGUISupportQt.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Frameworks/QtWidgets.framework/Versions/5/QtWidgets (compatibility version 5.8.0, current version 5.8.0)
    @executable_path/../Frameworks/QtGui.framework/Versions/5/QtGui (compatibility version 5.8.0, current version 5.8.0)
    @executable_path/../Libraries/libvtkFiltersGeneric.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkFiltersHyperTree.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkIOExportOpenGL2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkIOExport.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkRenderingGL2PSOpenGL2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkRenderingMatplotlib.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkRenderingParallel.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkRenderingVolumeAMR.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkFiltersAMR.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkRenderingVolumeOpenGL2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkRenderingOpenGL2.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    /System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 20.0.0)
    @executable_path/../Libraries/libvtkImagingMath.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkglew.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkFiltersParallelStatistics.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkIOEnSight.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkIOImport.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkIOParallel.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkIOGeometry.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkIONetCDF.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkIOParallelExodus.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkIOExodus.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkexoIIc.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkIOParallelXML.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkInteractionWidgets.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkInteractionStyle.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkRenderingAnnotation.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkRenderingVolume.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkImagingSources.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkChartsCore.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkRenderingContext2D.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkRenderingFreeType.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkfreetype.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    /System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices (compatibility version 1.0.0, current version 48.0.0)
    /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 59.0.0)
    @executable_path/../Libraries/libvtkNetCDF_cxx.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkNetCDF.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkhdf5_hl.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkhdf5.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkIOXML.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)
    @executable_path/../Libraries/libvtkPVServerManagerApplication.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkPVServerManagerCore.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkjsoncpp.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkpugixml.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkPVServerImplementationCore.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkPVClientServerCoreCore.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkPVVTKExtensionsCore.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkIOImage.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkFiltersParallel.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkRenderingCore.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkFiltersExtraction.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkFiltersStatistics.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkImagingFourier.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkImagingCore.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkalglib.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkParallelCore.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkIOLegacy.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkFiltersGeometry.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkFiltersModeling.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkFiltersSources.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkFiltersGeneral.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkFiltersCore.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkCommonComputationalGeometry.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkFiltersProgrammable.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkPVCommon.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkIOXMLParser.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkIOCore.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkCommonExecutionModel.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkCommonDataModel.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkCommonTransforms.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkCommonSystem.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkCommonMisc.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkCommonMath.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libprotobuf.dylib (compatibility version 0.0.0, current version 0.0.0)
    @executable_path/../Libraries/libvtkClientServer.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkPythonInterpreter.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkWrappingPython36Core.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkCommonCore.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libtbb.dylib (compatibility version 0.0.0, current version 0.0.0)
    @executable_path/../Libraries/libvtksys.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libpython3.6m.dylib (compatibility version 3.6.0, current version 3.6.0)
    @executable_path/../Libraries/libvtktiff.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkzlib.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Libraries/libvtkjpeg.1.dylib (compatibility version 1.0.0, current version 1.0.0)
    @executable_path/../Frameworks/QtNetwork.framework/Versions/5/QtNetwork (compatibility version 5.8.0, current version 5.8.0)
    @executable_path/../Frameworks/QtCore.framework/Versions/5/QtCore (compatibility version 5.8.0, current version 5.8.0)
    /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)

I don't see any smoking guns 🚬 🔫

[mathturtle]

I don't see any smoking guns 🚬 🔫

Neither do I... what OSX version are you running Cory? Maybe the minimum OSX version is bad on some library somewhere in the bundle.

[cquammen]

Nothing too ancient :-)

[mathturtle]

Then that could be the problem. The build machine is running 10.12. I think otool -l shows the minimum OSX version somewhere in its output, but you'd have to run it on each library to find the bad one...

[cquammen]

Attached is the output of running

ls *.dylib | xargs -I {} sh -c 'echo {}; otool -l {} | grep LC_VERSION_MIN_MACOSX --after-context=4 | grep version'

from

/Applications/tomviz.app/Contents/Libraries

mac_min_version.txt

Looks like most libraries support 10.9. The following require 10.10:

libgcc_s.1.dylib
  version 10.10
--
libgcc_s_ppc64.1.dylib
  version 10.10
--
libgcc_s_x86_64.1.dylib
  version 10.10
--
libgfortran.3.dylib
  version 10.10
--
libgfortran.dylib
  version 10.10
--
libquadmath.0.dylib
  version 10.10
--
libquadmath.dylib
  version 10.10

Some support 10.12:

libavcodec.55.69.100.dylib
  version 10.12
--
libavcodec.55.dylib
  version 10.12
--
libavcodec.dylib
  version 10.12
--
libavformat.55.48.100.dylib
  version 10.12
--
libavformat.55.dylib
  version 10.12
--
libavformat.dylib
  version 10.12
--
libavutil.52.92.100.dylib
  version 10.12
--
libavutil.52.dylib
  version 10.12
--
libavutil.dylib
  version 10.12
--
libpython3.6m.dylib
  version 10.12
--
libswscale.2.6.100.dylib
  version 10.12
--
libswscale.2.dylib
  version 10.12
--
libswscale.dylib
  version 10.12

None require 10.11.

[cquammen]

Ah, and there it is. libpython3.6m.dylib won't run on my 10.11. Please don't make me upgrade to 10.12 🙏

[cjh1]

We are building with 10.9 sysroot and min version of 10.9, so I'm not sure why it requires 10.12 :frowning:

[mathturtle]

I knew about the ones that require 10.10... I didn't have a 10.9 system to get 10.9 versions of them and I had to copy the old ones off of bigmac before we upgraded it to 10.12. So that is our minimum version :(. No one has complained though (even though the website still claims we support 10.9, it lies). The others are potential problems (and although I think the libpython3.6m.dylib is probably the culprit we should try to fix all of them).

[cjh1]

OK, I have found it ( by digging into the configure command ). -mmacosx-version-min=10.9 is not enough we also need to set a environment variable MACOSX_DEPLOYMENT_TARGET ...

[cjh1]

Merged change into superbuild :crossed_fingers:

[cjh1]

Those other 10.12 dylib's are coming from ffmpeg

[cjh1]

Looks like others have run into the same issue: http://ffmpeg.gusari.org/viewtopic.php?f=8&t=1736

[cjh1]

It might be worth trying passing the -mmacosx-version-min=10.9 option to -extra-cflags="..." is case its overriding the CFLAGS environment variable.

[cjh1]

Scratch that, the paraview-superbuild is doing that, but has the same issue ...

[cjh1]

@cquammen Could you try out the latest: https://open.cdash.org/viewFiles.php?buildid=4819432

[cquammen]

No go. Crash with the same call stack as reported initially.

[cjh1]

@cquammen can you check libpython3.6m.dylib with your otool magic?

[cquammen]

Good news: Python no longer requires 10.12, but the following still do:

libavcodec.55.69.100.dylib
  version 10.12
--
libavcodec.55.dylib
  version 10.12
--
libavcodec.dylib
  version 10.12
--
libavformat.55.48.100.dylib
  version 10.12
--
libavformat.55.dylib
  version 10.12
--
libavformat.dylib
  version 10.12
--
libavutil.52.92.100.dylib
  version 10.12
--
libavutil.52.dylib
  version 10.12
--
libavutil.dylib
  version 10.12
--
libswscale.2.6.100.dylib
  version 10.12
--
libswscale.2.dylib
  version 10.12
--
libswscale.dylib
  version 10.12

libpython is down to

libpython3.6m.dylib
  version 10.9

[cjh1]

The ffmpeg ones shouldn't be a problem as paraview has the same problem, but paraview works on you machine ( I assume! ) ...

[cryos]

It may not when he attempts to export a movie... I think we can live with that for now, if everything else is working and at the end of the day Mac users can upgrade for free :apple:

[cquammen]

Yep ParaView runs fine - you can't even save out an MPEG on the Mac in either ParaView or Tomviz. I'm not sure why we ship those with ParaView - probably an oversight.

[cryos]

If you upgrade to Sierra maybe you can? We got some additional clues on Bob's Mac,

Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: Py_Initialize: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'

So that led to some new ideas on missing files, hopefully the next build will be better. Running the application from the command line showed those errors, but I didn't see them in the error dialog from macOS.

[cryos]

This should be working now, if you get a chance @cquammen we would appreciate a few extra data points. If you see a crash please try running on the command line to capture extra debug output from Python.

[cquammen]

I'm happy to try it if you point me to the binary.

[cryos]

The nightly (which isn't really just nightly) from http://www.tomviz.org/downloads/ should work.

[cquammen]

Roger that.

I get the same errors when run from the command line:

Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: Py_Initialize: unable to load the file system codec
ModuleNotFoundError: No module named 'encodings'

[cquammen]

FWIW, the directory /Applications/tomviz.app/Contents/Python/encodings exists in the installed bundle.

[cquammen]

If in my terminal I set

export PYTHONPATH=/Applications/tomviz.app/Contents/Python/

and launch Tomviz, Python works! I still get the following messages:

Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]

but at least 🐍 works.

[cjh1]

I am working at setting PYTHONHOME, to fix that up.

[cryos]

Just reproduced here, I think the shell we used in testing earlier still had PYTHONPATH exported, fresh shell it fails until I set PYTHONPATH. Thanks for catching that @cquammen!

[cryos]

I think this perhaps explains why it is not working, some of these paths must be available while Python is being initialized,

/**
 * Prepends the path to the sys.path variable. If Python has been
 * initialized, this call will update the sys.path variable otherwise the same
 * will be done once Python has been initialized. The paths added are saved so
 * that if Python is initialized again (by calls to Initialize()), then these
 * paths will be re-added.
 */

This is the function we call, I am going to try something a little crazy to see if it works setting the environment variable for our running application.

[cjh1]

The call to InitializePythonEnvironment(...) occurs from main(...) which is before we init our Python interrupter so I would expect this to work :frowning:

[cjh1]

Actually, looking at the implementation on vtkPythonAppInitPrependPythonPath, python is definitely already running. As @cryos pointed out its append to sys.path directly. Lets hope the qputenv approach works.

[cryos]

It did fail before, and work in my test. I will reboot the Mac in a minute to be as clean as possible, but I think we are good...

[cryos]

This is fixed in the latest binaries, 0.9.4-220 or later should work without issue.

[cquammen]

confirmed that this works on my machine. High five.

[cryos]

That's great - thanks for the confirmation!