Mac: can't run because 'ImportError: cannot import name decode_dss_signature'

[joerick]

This bug report from John Batty on KS

Just tried installing the latest Tide on my Mac (Tide-0.3.3.dmg). It loads up with an example. If I press play (to run it on simulator) it fails (details below) :-( Anyone else seeing this? Exception...

Exception...

Traceback (most recent call last):                                                                                                                            
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy                                                                          
.py", line 162, in _run_module_as_main                                                                                                                        
    "__main__", fname, loader, pkg_name)                                                                                                                      
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy                                                                          
.py", line 72, in _run_code                                                                                                                                   
    exec code in run_globals                                                                                                                                  
  File "/Applications/Tide.app/Contents/Resources/vendor/tide-packages/tbtool/__main                                                                          
__.py", line 5, in [module]                                                                                                                               
    import paramiko                                                                                                                                           
  File "/Applications/Tide.app/Contents/Resources/vendor/tide-packages/paramiko/__in                                                                          
it__.py", line 30, in [module]                                                                                                                          
    from paramiko.transport import SecurityOptions, Transport                                                                                                 
  File "/Applications/Tide.app/Contents/Resources/vendor/tide-packages/paramiko/tran                                                                          
sport.py", line 53, in [module]                                                                                                                             
    from paramiko.dsskey import DSSKey                                                                                                                        
  File "/Applications/Tide.app/Contents/Resources/vendor/tide-packages/paramiko/dssk                                                                          
ey.py", line 27, in [module]                                                                                                                                  
    from cryptography.hazmat.primitives.asymmetric.utils import (                                                                                             
ImportError: cannot import name decode_dss_signature                                                                                                          

Process exited with code 1.

[joerick]

So I wonder if Python is trying to import a different version of cryptography - do you know if you have another version installed on your system?

[joerick]

Could you copy+paste this into a new app in Tide and then tell me the output?

import cryptography

print 'Version:', cryptography.__version__
print 'File:', cryptography.__file__

(Mine looks like: )

[xtramural]

My output is:

Version: 1.4                                                                                                  
File: /Applications/Tide.app/Contents/Resources/vendor/tide-packages/cryptography/__init__.pyc                                                                                                    

Process exited.  

[johnbatty]

Hmm... Unfortunately printing out the cryptography version from within Tide fails in exactly the same way. From the stack trace it looks like it all stems from Tide importing paramiko - presumably before starting to run the program.

If I run the code from a standard (non-Tide) Python app then I get this...

john$ python test.py
Version: 0.8.2
File: /Library/Python/2.7/site-packages/cryptography-0.8.2-py2.7-macosx-10.10-intel.egg/cryptography/__init__.pyc

So this looks much older than yours. So could possibly be picking this up, I guess. I'll have a play and see if I can dump out the version from within Tide...

[joerick]

Ah, yes, the error is actually coming from the tool (tbtool) that starts the app! Damn. Somehow your system-wide version must be getting picked up before our bundled version.

How is Python installed on your system? Did you install it manually or via Homebrew? What's the output of which python?

[johnbatty]

So, I just modified the Tide Python code to dump out info before loading paramiko...

[/Applications/Tide.app/Contents/Resources/vendor/tide-packages/tbtool/__main__.py]
#!/usr/bin/env python

from docopt import docent
import os, textwrap, shutil, filecmp, subprocess, sys, logging, fnmatch

print("sys.path=%s" % sys.path)
print("PYTHONPATH=%s" % os.environ['PYTHONPATH'])
import cryptography
print("cryptography version: %s" % cryptography.__version__)
print("cryptography file: %s" % cryptography.__file__)

import paramiko

Here's the output...

sys.path=['', '/Library/Python/2.7/site-packages/python_etcd-0.3.3-py2.7.egg', '/Lib
rary/Python/2.7/site-packages/pyOpenSSL-0.15.1-py2.7.egg', '/Library/Python/2.7/site
-packages/urllib3-1.10.4-py2.7.egg', '/Library/Python/2.7/site-packages/cryptography
-0.8.2-py2.7-macosx-10.10-intel.egg', '/Library/Python/2.7/site-packages/cffi-1.0.0b
2-py2.7-macosx-10.10-intel.egg', '/Library/Python/2.7/site-packages/enum34-1.0.4-py2
.7.egg', '/Library/Python/2.7/site-packages/pyasn1-0.1.7-py2.7.egg', '/Library/Pytho
n/2.7/site-packages/pycparser-2.12-py2.7.egg', '/Library/Python/2.7/site-packages/Ch
erryPy-3.8.0-py2.7.egg', '/Library/Python/2.7/site-packages/bpython-0.14.2-py2.7.egg
', '/Library/Python/2.7/site-packages/greenlet-0.4.9-py2.7-macosx-10.11-intel.egg', 
'/Library/Python/2.7/site-packages/curtsies-0.1.19-py2.7.egg', '/Library/Python/2.7/
site-packages/blessings-1.6-py2.7.egg', '/Library/Python/2.7/site-packages/Adafruit_
GPIO-0.9.3-py2.7.egg', '/Users/john/Documents/modulo/modulo-python', '/Library/Pytho
n/2.7/site-packages', '/Applications/Tide.app/Contents/Resources/vendor/tide-package
s', '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip', '/S
ystem/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7', '/System/Libr
ary/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin', '/System/Li
brary/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac', '/System/Lib
rary/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpacka
ges', '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', 
'/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk', '/S
ystem/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old', '/Sys
tem/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload', '/U
sers/john/Library/Python/2.7/lib/python/site-packages', '/usr/local/lib/python2.7/si
te-packages', '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/p
ython/PyObjC', '/Library/Python/2.7/site-packages']                                 
PYTHONPATH=/Applications/Tide.app/Contents/Resources/vendor/tide-packages           
cryptography version: 0.8.2                                                         
cryptography file: /Library/Python/2.7/site-packages/cryptography-0.8.2-py2.7-macosx
-10.10-intel.egg/cryptography/__init__.pyc

So this confirms that Tide is picking up my older cryptography module. And that my sys.path is pretty crazy.

I honestly couldn't tell you how I installed Python - I did it so long ago. Could have been manually, or via Homebrew, or via "port"... Or possibly a combination! :-(

john$ which python
/usr/bin/python

Looks like I need to get the Tide path further up the sys.path... It is possible to hack sys.path directly. I'll try this out as a workaround. However this is obviously pretty ugly. May need to look at using virtualenv or similar?

[johnbatty]

I can confirm that hacking sys.path in main.py fixes the problem...

[/Applications/Tide.app/Contents/Resources/vendor/tide-packages/tbtool/__main__.py]
#!/usr/bin/env python

from docopt import docent
import os, textwrap, shutil, filecmp, subprocess, sys, logging, fnmatch

sys.path.insert(1, '/Applications/Tide.app/Contents/Resources/vendor/tide-packages')

[joerick]

Excellent debugging! Thats... pretty crazy! There must be something on your system putting those import paths before PYTHONPATH. I wonder if this is .pth files in your installation.

Virtualenvs can be used to run the app in, but unfortunately the tool (tbtool) that builds those virtualenvs imports paramiko!

[xtramural]

That's good news John. Thanks for helping to debug the issue.

[joerick]

Looks like the culprit are eggs installed using easy_install - http://stackoverflow.com/questions/5984523/eggs-in-path-before-pythonpath-environment-variable

[joerick]

Apparently you can remove using pip-

sudo pip uninstall cryptography

You could even then do

sudo pip install cryptography

To restore that module if you still need it. But pip will install it as a 'wheel' not an 'egg', so we shouldn't have these problems.

Ahh... python packaging...

[benpawle]

Nice work chaps this is great!

[johnbatty]

Thanks for the suggestion @joerick, but that would be too easy wouldn't it... :-) pip claims to know nothing about my cryptography module...

Johns-MacBook-Pro-2:~ john$ sudo -H pip list
click (6.6)
connexion (1.0.97)
Flask (0.11)
itsdangerous (0.24)
Jinja2 (2.8)
jsonschema (2.5.1)
MarkupSafe (0.23)
pip (8.1.2)
PyYAML (3.11)
requests (2.10.0)
setuptools (21.2.2)
six (1.10.0)
strict-rfc3339 (0.7)
swagger-spec-validator (2.0.2)
Werkzeug (0.11.10)
wheel (0.26.0)
Johns-MacBook-Pro-2:~ john$ sudo -H pip uninstall cryptography
Cannot uninstall requirement cryptography, not installed

I can see the cryptography module in the easy-install.ph though...

Johns-MacBook-Pro-2:~ john$ cat /Library/Python/2.7/site-packages/easy-install.pth
import sys; sys.__plen = len(sys.path)
./python_etcd-0.3.3-py2.7.egg
./pyOpenSSL-0.15.1-py2.7.egg
./urllib3-1.10.4-py2.7.egg
./cryptography-0.8.2-py2.7-macosx-10.10-intel.egg
./cffi-1.0.0b2-py2.7-macosx-10.10-intel.egg
./enum34-1.0.4-py2.7.egg
./pyasn1-0.1.7-py2.7.egg
./pycparser-2.12-py2.7.egg
./CherryPy-3.8.0-py2.7.egg
./bpython-0.14.2-py2.7.egg
./greenlet-0.4.9-py2.7-macosx-10.11-intel.egg
./curtsies-0.1.19-py2.7.egg
./blessings-1.6-py2.7.egg
./Adafruit_GPIO-0.9.3-py2.7.egg
/Users/john/Documents/modulo/modulo-python
/Library/Python/2.7/site-packages
import sys; new=sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p]=new; sys.__egginsert = p+len(new)

Looking at the dates on the packages I can see it was probably installing python_etcd or urllib3 that either pulled it in, or got me to install it...

Johns-MacBook-Pro-2:~ john$ ls -ltr /Library/Python/2.7/site-packages/
total 11032
-rw-r--r--   1 root  wheel    41481  3 Sep  2014 pathlib.py
drwxr-xr-x   9 root  wheel      306 10 Mar  2015 pyflakes-0.8.1.dist-info
drwxr-xr-x  16 root  wheel      544 10 Mar  2015 pyflakes
...
-rw-r--r--   1 root  wheel    23862  6 May  2015 docopt.pyc
drwxr-xr-x   7 root  wheel      238  6 May  2015 docopt-0.6.2-py2.7.egg-info
drwxr-xr-x   4 root  wheel      136  8 May  2015 python_etcd-0.3.3-py2.7.egg
-rw-r--r--   1 root  wheel   144776  8 May  2015 urllib3-1.10.4-py2.7.egg
-rw-r--r--   1 root  wheel   235456  8 May  2015 pyOpenSSL-0.15.1-py2.7.egg
drwxr-xr-x   4 root  wheel      136  8 May  2015 cryptography-0.8.2-py2.7-macosx-10.10-intel.egg
-rw-r--r--   1 root  wheel    78047  8 May  2015 enum34-1.0.4-py2.7.egg
drwxr-xr-x   8 root  wheel      272  8 May  2015 cffi-1.0.0b2-py2.7-macosx-10.10-intel.egg
-rw-r--r--   1 root  wheel   201637  8 May  2015 pycparser-2.12-py2.7.egg
-rw-r--r--   1 root  wheel    79578  8 May  2015 pyasn1-0.1.7-py2.7.egg
-rw-r--r--   1 root  wheel       58 19 May  2015 readline.py
-rwxr-xr-x   1 root  wheel   690516 19 May  2015 gnureadline.so
...
drwxr-xr-x  48 root  wheel     1632 30 May 14:40 jinja2
drwxr-xr-x  10 root  wheel      340 30 May 14:40 Jinja2-2.8.dist-info

I've fixed / worked around for now by simply manually removing all the modules from 8 May from my easy-install.pth. I then reinstalled Tide to get a clean version and it ran fine. Marvellous!

So I'm happy now. But you may still want to think about how you can fix "properly". But I guess you can wait to see if anyone else hits it... My setup is probably "out of the ordinary"!

[joerick]

Cool, glad you've got that working. Thank you for documenting the the problem so well here so hopefully others will find it too. What with Fink, MacPorts, Homebrew, easy_install, pip, wheels and eggs over the years, I'm sure you won't be the last with Python environment issues!

There might be a way to resolve this by bundling a full install of Python with Tide (that's how it works on Windows) but I ran into some trouble down that path last I tried, so we've got the PYTHONPATH solution for now.

[joerick]

This is fixed in f202cccee63cd3764abfb5cf4e5ec67133fcf861, by using the sizecustomize module to hack around the problem!