Can we use python3 on Mac?

[dkp]

Dear Chris,

I am trying to use Surfice with Python3 on a Mac. I set up a Python3 Conda environment. I tried pointing the surfice.ini (in /Users/dpat/Library/Application Support/surfice) to my Conda Python3 installation, but Surfice scripts failed when they started up.

Is there anything I can do to use Python 3 for scripting Surfice on Mac?

[neurolabusc]

I assume you are using the latest release. For macOS this will be v1.0.20201102++ (with the ++ indicating improved M1 support).

1. Choose the Preferences menu item
2. Press the Advanced button
3. Your preferences text file opens. Set the PyLib to point to your desired libpython*.dylib. For me the default location was

   PyLib=/System/Library/Frameworks/Python.framework/Versions/Current/lib/libpython2.7.dylib

   But my newer version is at:

   PyLib=/Users/chrisrorden/miniforge3/lib/libpython3.9.dylib 

4. Restart Surfice. I would run a minimal script to ensure all is well:

   import gl
   import sys
   print(sys.version)

   For me, this reports

   Running Python script
   3.9.2 | packaged by conda-forge | (default, Feb 21 2021, 05:02:58) 
   [Clang 11.0.1 ]

[dkp]

Hi Chris,

Thanks for your instructions. I am running Surf Ice v1.0.20201102++ 64-bit x86-64 Cocoa LLVM, which appears to be the latest version. The OS is Catalina. I created a conda environment with Python 3.9.2. I tried running from both within the Conda environment and outside the environment.

From within the environment:

$ which python  
/Users/dpat/anaconda3/envs/surfice/bin/python  

$ python --version  
Python 3.9.2  

Then, I created a simple test script:

import gl  
import sys  
print(sys.version)  

and ran it:

$ surfice -S rtest.py   
Loading preferences: /Users/dpat/Library/Application Support/surfice/surfice.ini  
Using PyLib from preferences "/Users/dpat/anaconda3/envs/surfice/lib/libpython3.9.dylib"  
Error: Could not open Dll "libpython3.9.dylib"  
Python could not be properly initialized. We must quit.  

Exiting the Conda environment and re-running it does not help (I get the same error).

Any insights would be appreciated: is this a permission issue? wrong DLL? your version works because you're running in miniforge? Thanks for your assistance in getting this working.

[neurolabusc]

Surfice is distributed as a notarized application, and recent MacOS releases have changed security measures substantially. The behavior your are seeing suggests that the dylib file exists, but the operating system's dlopen() function refuses to load it. This suggests some security measure is blocking this. I would try a couple things:

1. Check the LC_VERSION_MIN_MACOSX of your library, e.g. otool -l /Users/dpat/anaconda3/envs/surfice/lib/libpython3.9.dylib | grep -B 1 -A 3 LC_VERSION_MIN
2. Check if the Security and Privacy System Preferences reveals anything.

I also note that Apple Catalina release notes suggests If your software depends on scripting languages, it’s recommended that you bundle the runtime within the app. Therefore, I guess the long term solution for this is for future releases of Surfice to include the scripting language inside the notarized app bundle. This will require a lot more disk space, and users will not be able to change the scripting code, as the notarized application would no longer be valid:

$codesign -vvvv /Applications/Surfice/surfice.app/Contents/MacOS/surfice 
--prepared:/Applications/Surfice/surfice.app/Contents/MacOS.lipo
--validated:/Applications/Surfice/surfice.app/Contents/MacOS.lipo
/Applications/Surfice/surfice.app/Contents/MacOS/surfice: a sealed resource is missing or invalid
 file added: /Applications/Surfice/surfice.app/Contents/Resources/libpython3.9.dylib

So my guess is that for the current version of Surfice on Catalina, you will have to live Python 2.7 provided with the system, and future versions fo Surfice for MacOS will include a specific version of Python that you will be unable to change.

I discuss some of the other challenges scientists face with Apple's recent security measures here.

[dkp]

Hi Chris,

Thank you for putting us onto the correct path to resolve the problem!

The solution is to turn off SIP:
https://www.imore.com/how-turn-system-integrity-protection-macos

Now it all works flawlessly! Yay!

-Dianne

[neurolabusc]

Your experiment demonstrates SIP is blocking Python3 usage in macOS Catalina. However, I would not recommend disabling SIP on a regular basis. I would suggest making sure your code is Python 2.7 compatible, which will allow you to use the default installation. I do expect the next release of Surfice will come with Python 3.x support built in. However, I want to wait for Python to stabilize on the new Apple M1 CPUs before pushing that next release. The latest version of Surfice natively supports both x86-64 and Apple Silicon M1 CPUs. This means I will probably need to create a universal Python to support these systems.