Freezing With py2app: IOError: could not get source code

[riggsd]

I'm attempting to "freeze" an application into a distributable OS X .app using py2app. I have slimmed my code down to the bare minimum which triggers this issue, but I don't understand enough of the magic to determine if the problem is truly with sounddevice, CFFI, py2app, or my configuration. Apologies if this is the wrong channel to start looking for a fix!

Once frozen, a simple import sounddevice causes the blowup. The exception is thrown when a C header is fed to CFFI as a giant string literal:

Traceback (most recent call last):
  File "sdtest/dist/sdtest.app/Contents/Resources/__boot__.py", line 77, in <module>
    _run()
  File "sdtest/dist/sdtest.app/Contents/Resources/__boot__.py", line 62, in _run
    exec(compile(source, path, 'exec'), globals(), globals())
  File "sdtest/dist/sdtest.app/Contents/Resources/sdtest.py", line 1, in <module>
    import sounddevice
  File "sounddevice.pyc", line 313, in <module>
  File "cffi/api.pyc", line 105, in cdef
  File "cffi/api.pyc", line 119, in _cdef
  File "cffi/cparser.pyc", line 299, in parse
  File "cffi/cparser.pyc", line 304, in _internal_parse
  File "cffi/cparser.pyc", line 260, in _parse
  File "cffi/cparser.pyc", line 40, in _get_parser
  File "pycparser/c_parser.pyc", line 116, in __init__
  File "pycparser/ply/yacc.pyc", line 3293, in yacc
  File "pycparser/ply/yacc.pyc", line 2938, in validate_all
  File "pycparser/ply/yacc.pyc", line 2982, in validate_modules
  File "inspect.pyc", line 690, in getsourcelines
  File "inspect.pyc", line 538, in findsource
IOError: could not get source code

My setup.py with py2app configuration is as follows; I've explicitly included the path to my compiled PortAudio lib (which magically installed with sounddevice) to ensure it is included within the the .app:

from setuptools import setup

PY2APP_OPTIONS = {
    'argv_emulation': False,
    'includes': ['sounddevice'],
    'frameworks': ['/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/_sounddevice_data/libportaudio.dylib'],
}

setup(
    app=['sdtest.py'],
    options={'py2app': PY2APP_OPTIONS},
    setup_requires=['py2app'],
)

And the sdtest.py file itself need only be as simple as this to trigger this problem:

import sounddevice

Can you provide any suggestions as to how better to track down the problem here? I don't really understand what CFFI is doing under the hood, nor understand "who" is unable to find source code nor what source it's looking for.

[riggsd]

Contents of the built sdtest.app, in the hopes that you may spot something important which is missing. Note that site-packages get rolled up into a .zip file, which can make clever imports fail:

.
./sdtest.app
./sdtest.app/Contents
./sdtest.app/Contents/Frameworks
./sdtest.app/Contents/Frameworks/libportaudio.dylib
./sdtest.app/Contents/Frameworks/Python.framework
./sdtest.app/Contents/Frameworks/Python.framework/Python
./sdtest.app/Contents/Frameworks/Python.framework/Resources
./sdtest.app/Contents/Frameworks/Python.framework/Versions
./sdtest.app/Contents/Frameworks/Python.framework/Versions/2.7
./sdtest.app/Contents/Frameworks/Python.framework/Versions/2.7/include
./sdtest.app/Contents/Frameworks/Python.framework/Versions/2.7/include/python2.7
./sdtest.app/Contents/Frameworks/Python.framework/Versions/2.7/include/python2.7/pyconfig.h
./sdtest.app/Contents/Frameworks/Python.framework/Versions/2.7/lib
./sdtest.app/Contents/Frameworks/Python.framework/Versions/2.7/lib/python2.7
./sdtest.app/Contents/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config
./sdtest.app/Contents/Frameworks/Python.framework/Versions/2.7/Python
./sdtest.app/Contents/Frameworks/Python.framework/Versions/2.7/Resources
./sdtest.app/Contents/Frameworks/Python.framework/Versions/2.7/Resources/Info.plist
./sdtest.app/Contents/Frameworks/Python.framework/Versions/Current
./sdtest.app/Contents/Info.plist
./sdtest.app/Contents/MacOS
./sdtest.app/Contents/MacOS/python
./sdtest.app/Contents/MacOS/sdtest
./sdtest.app/Contents/PkgInfo
./sdtest.app/Contents/Resources
./sdtest.app/Contents/Resources/__boot__.py
./sdtest.app/Contents/Resources/__error__.sh
./sdtest.app/Contents/Resources/include
./sdtest.app/Contents/Resources/include/python2.7
./sdtest.app/Contents/Resources/include/python2.7/pyconfig.h
./sdtest.app/Contents/Resources/lib
./sdtest.app/Contents/Resources/lib/python2.7
./sdtest.app/Contents/Resources/lib/python2.7/config
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/_AE.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/_bisect.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/_cffi_backend.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/_collections.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/_Ctl.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/_ctypes.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/_Dlg.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/_Evt.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/_File.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/_functools.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/_hashlib.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/_heapq.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/_io.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/_locale.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/_Menu.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/_multibytecodec.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/_Qd.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/_random.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/_Res.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/_scproxy.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/_socket.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/_ssl.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/_struct.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/_Win.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/array.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/binascii.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/bz2.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/cPickle.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/cStringIO.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/datetime.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/fcntl.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/gestalt.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/grp.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/itertools.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/MacOS.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/math.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/Nav.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/operator.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/pyexpat.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/resource.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/select.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/strop.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/termios.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/time.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/unicodedata.so
./sdtest.app/Contents/Resources/lib/python2.7/lib-dynload/zlib.so
./sdtest.app/Contents/Resources/lib/python2.7/site-packages.zip
./sdtest.app/Contents/Resources/lib/python2.7/site.pyc
./sdtest.app/Contents/Resources/PythonApplet.icns
./sdtest.app/Contents/Resources/sdtest.py
./sdtest.app/Contents/Resources/site.pyc

[mgeier]

Sorry, I have no clue. You should probably ask on the CFFI mailing list: https://groups.google.com/forum/#!forum/python-cffi.

Once you found a solution, please report it back. It might be interesting for other users!

[arigo]

According to the traceback, the problem is in pycparser or its submodule ply. Maybe something like lextab.py/yacctab.py have not been pregenerated, or the check for if they have been is broken when there are only .pyc files around and no .py files. You should try asking on the pycparser mailing list.

(Note that this error would not occur if you use the API mode, but only in the ABI mode of CFFI.)

[riggsd]

Update: The bug is in an old version of PLY (version 3.8) which has been bundled into pycparser source tree. Upstream PLY (version 3.9) is already fixed. I have filed an issue with pycparser requesting that they update their copied dependency.

See: https://github.com/eliben/pycparser/issues/160

[riggsd]

Update: This still affects python-sounddevice because it calls CFFI in such a way that pycparser is invoked, and pycparser has not yet released an updated version with the fixed ply module (though pycparser's repository HEAD is fixed).

[mgeier]

@riggsd Thanks for the update! I'll leave this open for now.

[peircej]

I've been running into this problem for the PsychoPy package too (also distributing a mac version using py2app) and for that I was able to get around it by adding 'pycparser', to the packages list in my py2app setup dict (I actually added 'cffi' too but I think that wasn't strictly necessary).

This means that the py files get copied as well as the pyc files and the source can be found avoiding the source of the bug in pycparser.