cannot import nupic.bindings.math [OS X 10.10 Homebrew Python]

[rhuairahrighairidh]

I'm having problems installing nupic on a new OS X Yosemite mac. It appears to install correctly but then trying to import the bindings crashes python.

>>> import nupic.bindings.math
Fatal Python error: PyThreadState_Get: no current thread
Abort trap: 6

I'm using homebrew's version of python (v2.7.9), not the system version (v2.7.6). Here's the link to the mailing list thread http://lists.numenta.org/pipermail/nupic_lists.numenta.org/2015-January/010230.html

Here’s how I installed everything (on a new mac):

1. I installed XCode from the app store.
2. set ARCHFLAGS="-arch x86_64" in .bash_profile
3. I installed home-brew in the standard way ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
4. installed python 2.7.9 homebrew install python
5. installed virtualenv pip install virtualenv
6. set up a virtual env in called "all" in ~/dev/ virtualenv all
7. installed nupic dependencies brew install automake libtool cmake
8. activated my virtualenv source all/bin/activate
9. installed NuPIC pip install nupic

It seemed to install OK. But I get the error above.

• I have tried cloning the nupic repo and installing from that, but it gave the same error.
• I also tried cloning the nupic.core repo and installing that as well. But again the same error.
• Running which python outside of a virtualenv gives /usr/local/bin/python which is homebrew's python.

  Some Further Investigating:

I found the point at which python crashes trying to run import nupic.bindings.math is when it tries to import a file called _math.so. (a copy is in $NUPIC/nupic/bindings/ after install) Looking at the console error log thing there is a reference to two versions of python, one is Homebrew's the other is Apple's. I think the problem is that when _math.so is compiled it's somehow linked to the wrong version of python.

$ otool -L $NUPIC/nupic/bindings/_math.so 
~/dev/nupic/nupic/bindings/_math.so:
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
    /System/Library/Frameworks/Python.framework/Versions/2.7/Python (compatibility version 2.7.0, current version 2.7.6)
    /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)

Looking at the output whilst nupic installs I did spot some references to non Homebrew python. for example:

building 'nupic.bindings._math' extension
clang -fno-strict-aliasing -fno-common -dynamic -I/usr/local/include -I/usr/local/opt/sqlite/include -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk -I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -arch x86_64 -DNUPIC2 -DNTA_PLATFORM_darwin64 -DNTA_PYTHON_SUPPORT=2.7 -DNTA_INTERNAL -DNTA_ASSERTIONS_ON -DNTA_ASM -DHAVE_CONFIG_H -DBOOST_NO_WREGEX -I~/dev/nupic/external/darwin64/include -I~/dev/nupic/external/common/include -I~/dev/nupic/extensions -I~/dev/nupic -I~/dev/nupic.core/build/release/include -I~/dev/test/bin/../include/python2.7 -I~/dev/test/lib/python2.7/site-packages/numpy/core/include -I/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c extensions/py_support/NumpyVector.cpp -o build/temp.macosx-10.10-x86_64-2.7/extensions/py_support/NumpyVector.o -std=c++11 -m64 -fPIC -fvisibility=hidden -Wall -Wreturn-type -Wunused -Wno-unused-parameter -stdlib=libc++

Apparently -isysroot is set in the python.rb formula for Homebrew: https://github.com/Homebrew/homebrew/blob/master/Library/Formula/python.rb#L213

I've found a few references to similar problems online. Unfortunately none of them have any definitive answers short of replacing the system's python. https://stackoverflow.com/questions/20003268/impose-linking-to-specific-python-library-on-osx-lion https://stackoverflow.com/questions/22649379/swig-generated-code-links-to-wrong-python-installation https://stackoverflow.com/questions/2020228/swig-python-bindings-for-opencv-2-0-with-macports

[rhyolight]

@oxtopus Maybe you can help with this. It seems the problem may be that the python used when calling the setup is not the python being used within the build?

[cedricvidal]

Hi guys, for the record, I'm having the exact same problem :/

[rhyolight]

@cedricvidal Thanks for the additional data point. I'm going to have to find a fresh Mac to test out this situation.

[ghost]

Same problem for me too. brew doctor is ready to brew But when I try to run the swarm.py in the hotgym example I get the following error Fatal Python error: PyThreadState_Get: no current thread

However, I was able to compile from source, run all the tests successfully and run the hotgym example.

[rhuairahrighairidh]

@mellowg I've also tried compiling from source, but I still got the same error and test wouldn't run. Could you give us some more details on you did and your system setup?

[rhyolight]

I suspect it has something to do with having both the system Python and brew Python installed. I'll try to look into this more Monday.

---

Matt Taylor OS Community Flag-Bearer Numenta

On Sun, Feb 1, 2015 at 8:10 AM, rhuairahrighairidh <notifications@github.com

wrote:

@mellowg https://github.com/mellowg I've also tried compiling from source, but I still got the same error and test wouldn't run. Could you give us some more details on you did and your system setup?

— Reply to this email directly or view it on GitHub https://github.com/numenta/nupic/issues/1791#issuecomment-72370784.

[ghost]

@rhyolight is probably right. But I don't have enough experience to say exactly what is happening.

I ended up following most of the instructions at http://wolfpaulus.com/jounal/mac/installing_python_osx/, especially steps 4-6

Of course I replaced the path they use /System/Library/Frameworks/Python.framework/Versions/2.7/bin with /usr/local/bin so it would use the brewed Python

[rhyolight]

@mellowg Did it work?

---

Matt Taylor OS Community Flag-Bearer Numenta

On Sun, Feb 1, 2015 at 2:06 PM, mellowg notifications@github.com wrote:

@rhyolight https://github.com/rhyolight is probably right. But I don't have enough experience to say exactly what is happening.

I ended up following most of the instructions at http://wolfpaulus.com/jounal/mac/installing_python_osx/, especially steps 4-6

Of course I replaced the path they use /System/Library/Frameworks/Python.framework/Versions/2.7/bin with /usr/local/bin so it would use the brewed Python

— Reply to this email directly or view it on GitHub https://github.com/numenta/nupic/issues/1791#issuecomment-72387119.

[ghost]

@rhyolight In regards to compiling, yes it did work, and I can successfully run the examples. This is thanks to your clear instructions in the original mailing list thread (on 27 Jan 2015, at 23:58) about this issue. I suspect that some of steps I followed at http://wolfpaulus.com/jounal/mac/installing_python_osx/ created the proper environment for me to compile successfully.

But I could never get the pip-installed binary Nupic package working. I kept getting the same error @rhuairahrighairidh posted followed by a Python crash. So my success in compiling is not really a solution to this particular issue regarding the Nupic binary for Mac.

[rhyolight]

I'm getting a fresh Mac to test this condition today. I'll investigate tomorrow.

[rhyolight]

In the meantime, can one of your experiencing this error try building with the system python? You can uninstall the brew version with brew uninstall python. Let me know if it works or you get errors. Thank you.

[rhyolight]

I've replicated this error after installing brew python on this new Mac.

[rhyolight]

@david-ragazzi or @breznak Do you think this problem might have something to do with -isysroot being set to the system python location when building 'nupic.bindings._math' extension as @rhuairahrighairidh suggested in the original description?

[rhyolight]

I can replicate this error artificially on my primary development laptop quite easily:

1. using system python at /usr/bin/python, build and install nupic.core & nupic from source code
2. /usr/bin/python -c "import nupic.bindings.math" has no problems
3. brew install python (which installs to /usr/local/bin/python)
4. /usr/local/bin/python -c "import nupic.bindings.math" now throws the reported error

This tells us that this error is caused by nupic.core and/or nupic being built and/or installed using the system python, but run with the brew python.

It's the "ands" and "ors" in the previous statement that are causing me the most confusion. I have some more digging to do...

[rhyolight]

This definitely has something to do with the building 'nupic.bindings._math' extension logic flow. It's either this:

-isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk 

or this:

-I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers 

The funny thing is that I don't get these flags on my system when I build with brew python. I'm still trying to isolate exactly why and how these options are included when building with brew python.

[rhyolight]

@rhuairahrighairidh What version of setuptools do you have?

pip list | grep setuptools

[rhyolight]

Possibly relevant: http://stackoverflow.com/questions/7257760/how-to-make-python-and-distutils-use-the-right-sdk-on-mac-os-x

[rhyolight]

On my system (that works):

> python-config --cflags
-I/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/include/python2.7 
-I/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/include/python2.7 
-fno-strict-aliasing -fno-common -dynamic 
-I/usr/local/include 
-I/usr/local/opt/sqlite/include -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes

On my system that doesn't work:

> python-config --cflags
-I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 
-I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 
-fno-strict-aliasing -fno-common -dynamic -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes -DENABLE_DTRACE -arch x86_64

Notice the difference in the included python paths. Even though both python-config commands are being executed by the brew-installed python-config, the 2nd one is still including the system python paths in the --cflags.

[rhuairahrighairidh]

I think the flags -isysroot and the other -I one (refering to TK) are set when brew installs python. Here's the relevant lines in the python.rb formula: https://github.com/Homebrew/homebrew/blob/23c8efbdf46b9271a6a86928a25456683af52579/Library/Formula/python.rb#L211-221

Looks like they're both set if you don't(?) have the command line tools installed.

Here's some more information from my system

> pip list
pip (6.0.6)
setuptools (11.3.1)
virtualenv (12.0.5)

This is for brew python:

$ python-config --cflags
-I/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/include/python2.7
-I/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/include/python2.7
-fno-strict-aliasing -fno-common -dynamic
-I/usr/local/include
-I/usr/local/opt/sqlite/include
-isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk
-I/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.10.sdk/System/Library/Frameworks/Tk.framework/Versions/8.5/Headers
-DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes

and for the system python:

> /usr/bin/python-config --cflags
-I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7
-I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7
-fno-strict-aliasing -fno-common -dynamic -g -Os -pipe -fno-common -fno-strict-aliasing
-fwrapv -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wstrict-prototypes
-Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -Wstrict-prototypes
-DENABLE_DTRACE -arch x86_64

[rhyolight]

@rhuairahrighairidh Thanks for the help, and also for the wonderfully detailed bug report.

@cedricvidal & @mellowg Can you guys run python-config --cflags and show the output?

[rhyolight]

Potential solution (if the problem is only that XCode CLT were not installed before brew python):

1. brew uninstall python
2. install XCode command line tools
3. brew install python

To test, run python-config --cflags and ensure it does not include a system python location like -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7.

Please try this and tell me if it helps. I would test this out myself, but I'm trying to reinstall OS X on it from scratch.

[rhyolight]

My other macbook is reinstalling from scratch, which is apparently a 6 HOUR TASK! So I won't be able to test this thoroughly until tomorrow.

[rhuairahrighairidh]

I was doing a bit of investigating today. I was pretty sure that installing the CLT would fix it. However no such luck.

brew uninstall python --force xcode-select --install I had only Xcode installed before which apparently includes the tools, but in a different location. They still ran at the terminal. brew install python This time, as expected, python-config --cflags don't have any -isysroot

$ python-config --cflags
-I/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/include/python2.7
-I/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/include/python2.7
-fno-strict-aliasing -fno-common -dynamic
-I/usr/local/include
-I/usr/local/opt/sqlite/include
-DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes

$ python-config --ldflags
-L/usr/local/Cellar/python/2.7.9/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config
-lpython2.7 -ldl -framework CoreFoundation

Then I created a virtualenv and pip install nupic Everything seemed to go fine but then I still got the same error as before.

I also tried building nupic, but still got the same error. I built nupic.core as well for good measure but it didn't help.

_math.so still seems to be linked to the system python

$ otool -L $NUPIC/nupic/bindings/_math.so
/Users/ruaridh/dev/nupic/nupic/bindings/_math.so:
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1213.0.0)
    /System/Library/Frameworks/Python.framework/Versions/2.7/Python (compatibility version 2.7.0, current version 2.7.6)
    /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 120.0.0)

[rhyolight]

@rhuairahrighairidh Thanks for digging in more. I did some more investigation today too. See my comment here: https://github.com/numenta/nupic/issues/1813#issuecomment-73146036. I'm not so certain that the way that _math.so is being build by setuptools is the problem. Frankly, I'm stumped. I asked for some help on the nupic-hackers mailing list.

[dschwartz783]

One potential fix is to use a shim to redirect Python API calls to the appropriate version. A bit of a hack, but a practical one.

[dschwartz783]

Honestly, distributing binaries is always tricky. If you want to use a custom version of anything, you normally have to distribute it with the binaries, unless you can guarantee the user has the proper version installed at the correct path. The issue here is that once the libs are compiled, they're tied to the version of Python they compiled against. I suggest compiling different versions for different Python installations, or compiling against a shim that can handle redirecting the API calls to the appropriate version. Multiple versions is probably the simpler method. You could also just provide nupic via homebrew directly. That might be the most optimal solution. It would naturally be compiled against the homebrew version of Python.

Of course, if there are any Python modules on pypi that have c code and do something clever to overcome this issue, we may be able to borrow their code...

Incase anyone wasn't sure about what the error message meant: It can't get the current thread, because the version of python the library is linked to isn't the one that's running.

[rhyolight]

@res1233 Agreed. Thanks for your input. It seems crazy that python projects with C extensions must be built for OS X system python, homebrew python, and anaconda python even if they are all the same python version. Doesn't make sense to me. Does anyone know of a Python project with C extensions that distributes via pip without having to create different packages for each?

Seems like this should be a solved problem at this point for python projects.

[rhyolight]

@rhuairahrighairidh Installing from NuPIC source code is working for me on a fresh Mac with OS X 10.10.

Following your instructions from the issue description (sans virtualenv), with the last item changed to install from source code:

1. I installed XCode from the app store.
2. set ARCHFLAGS="-arch x86_64" in .bash_profile
3. I installed home-brew in the standard way ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
4. installed python 2.7.9 homebrew install python
5. installed nupic dependencies brew install automake libtool cmake
6. installed NuPIC cd $NUPIC; python setup.py install

This works, and python -c "import nupic.bindings.math" works, running NuPIC works, and swarming works.

This procedure downloads the nupic.core binaries from S3 (does not build nupic.core locally from source code), which were also built in the same Travis-CI environment that the NuPIC wheel was created.

I thought from previous discussion that we had a compilation problem as well, but that doesn't seem to be the case. I think this problem only manifests itself when attempting to install from a NuPIC binary wheel build on top of they Mac system python instead of homebrew.

[rhyolight]

@rhuairahrighairidh If you can try building from source using my instructions above and it works, I would like to close this and continue tracking the binary installation problem in #1813.

[rhyolight]

I am closing this and continue work on #1813.

[rhuairahrighairidh]

This is closed now but I installed as you asked (without virtualenv) and its all working. import nupic.bindings.math works and after installing mysql, all the tests pass.