search

cloudmatrix / esky

an auto-update framework for frozen python apps
BSD 3-Clause "New" or "Revised" License
362 stars 74 forks source link

unable to run bootrap file using py2app and python3.4 #104

Closed JPFrancoia closed 8 years ago

JPFrancoia commented 8 years ago

Hi again, I can successfully run a simple hello world program using py2app, esky and python 2, python 3 however fails..

Fatal Python error: Py_Initialize: unable to load the file system codec
ImportError: No module named 'encodings'

Running the second-level exe (not the boostrap one) works, the program runs smoothly. The issue seems to be related to #72.

JPFrancoia commented 8 years ago

I assumed python could not find some libraries, so I copied everything from the second level folder Contents into the first level folder Contents.

MyProgram.app/
    Info.plist
    Contents/
        MacOS/
            exe_esky
    MyProgram-mac-os-x86_64/
        MyProgram.app/
            Contents/

Now when I run exe_esky, I get this:

Failed to import the site module
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2226, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1200, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1129, in _exec
  File "<frozen importlib._bootstrap>", line 1467, in exec_module
  File "<frozen importlib._bootstrap>", line 1678, in get_code
  File "<frozen importlib._bootstrap>", line 656, in _compile_bytecode
ValueError: bad marshal data (unknown type code)
JPFrancoia commented 8 years ago

@rfk , any idea ? We are kind of stuck here :)

rfk commented 8 years ago

@JPFrancoia easiest might be to link or email me a tarball of the frozen app itself, that way I can poke around in its internals and see if I can see what's going on. I'll also try to repro locally.

rfk commented 8 years ago

(Wow, it's been a while since I actually tried to run this stuff myself)

With python3.4.3, py2app 0.9 I get the following error when trying to freeze an app:

AttributeError: 'ModuleGraph' object has no attribute 'scan_code'

Have you seen this before? Suggestion for a workaround?

JPFrancoia commented 8 years ago

Nope, but there is a fix here I guess: http://stackoverflow.com/questions/25394320/py2app-modulegraph-missing-scan-code

I'll email you that right away.

rfk commented 8 years ago

No luck on tracking down the problem with encodings module, but I suspect this subsequent error:

Failed to import the site module ... ValueError: bad marshal data (unknown type code)

May be because of the way we're writing bytecode in the py2app freezer here:

https://github.com/cloudmatrix/esky/blob/master/esky/bdist_esky/f_py2app.py#L138

Changing this to use the new esky.util.compile_to_bytecode function rather than calling marshal directly, may help things work after copying more resources into the top-level folder.

rfk commented 8 years ago

The encodings modules are in the zipfile at:

YourApp.app/YourApp-0.9.1.macosx-10_6-intel/YourApp.app/Contents/Resources/lib/python35.zip

Does copying this into the equivalent location in the top-level Contents directory help move things forward?

JPFrancoia commented 8 years ago

I had to copy both

YourApp.app/YourApp-0.9.1.macosx-10_6-intel/YourApp.app/Contents/Resources/lib/python35.zip

and

YourApp.app/YourApp-0.9.1.macosx-10_6-intel/YourApp.app/Contents/Resources/lib/python35/

Into the he equivalent location in the top-level Contents directory, otherwise I get this error:

Fatal Python error: Py_Initialize: unable to load the file system codec
zipimport.ZipImportError: can't decompress data; zlib not available
Current thread 0x00007fff7cd22000 (most recent call first):

When I do so, I go back to the error:

Failed to import the site module
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 969, in _find_and_load
  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 658, in exec_module
  File "<frozen importlib._bootstrap_external>", line 870, in get_code
  File "<frozen importlib._bootstrap_external>", line 473, in _compile_bytecode
ValueError: bad marshal data (unknown type code)

I tried to change the line:

f.write(marshal.dumps(compile("", "site.py", "exec")))

to 

f.write(esky.util.compile_to_bytecode("", "site.py"))

But I still get the bad marshal data error.

Did I made the proper modifications ? Do I have to make more ?

rfk commented 8 years ago

I tried to change the line: f.write(marshal.dumps(compile("", "site.py", "exec")))

It should replace both the f.write(imp.get_magic()) and f.write(marshal.dumps) lines.

JPFrancoia commented 8 years ago

Ok. I made the changes, but got:

Traceback (most recent call last):
  File "/Users/chembrows/Desktop/myprogram.app/dist/myrpogram.macosx-10_6-intel/myprogram.app/Contents/Resources/__boot__.py", line 695, in <module>
    sys.executable = environ["EXECUTABLEPATH"]
KeyError: 'EXECUTABLEPATH'
rfk commented 8 years ago

Hmm, I guess the behaviour of py2app must have changed in newer versions, either of it or of python3. To debug, I suggest replacing the _EXTRA_BOOTSTRAP_CODE section in f_py2app.py with the following:

_EXTRA_BOOTSTRAP_CODE = """
from posix import environ
print(list(environ.items()))
try:
  sys.executable = environ["EXECUTABLEPATH"]
  sys.argv[0] = environ["ARGVZERO"]
except KeyError:
  pass
"""

Which will hopefully (1) allow the bootstrapping exe to continue to load the real frozen app, and (2) tell us what's in environ so we might re-create the old behaviour.

JPFrancoia commented 8 years ago

Here we go:

Traceback (most recent call last):
[(b'PYTHONHOME', b'/Users/chembrows/Desktop/myprogram/dist/myprogram-0.9.1.macosx-10_6-intel/myprogram.app/Contents/Resources'), (b'TERM', b'xterm-256color'), (b'SHELL', b'/bin/bash'), (b'XPC_SERVICE_NAME', b'0'), (b'HISTFILESIZE', b'10000'), (b'LANG', b'fr_FR.UTF-8'), (b'LD_LIBRARY_PATH', b'/usr/lib/jvm/java-7-openjdk/jre/lib/amd64/'), (b'HOME', b'/Users/chembrows'), (b'_', b'./gui'), (b'ARGVZERO', b'./gui'), (b'PATH', b'/Library/Frameworks/Python.framework/Versions/3.5/bin:/opt/local/bin:/opt/local/sbin:/Library/Frameworks/Python.framework/Versions/3.5/bin:/Users/chembrows/Gabedit64:/Users/chembrows/Gabedit64:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/home/djipey/scripts_perso'), (b'DISPLAY', b'/private/tmp/com.apple.launchd.2p2Rzg1yiw/org.macosforge.xquartz:0'), (b'TERM_PROGRAM_VERSION', b'361'), (b'DBUS_LAUNCHD_SESSION_BUS_SOCKET', b'/private/tmp/com.apple.launchd.XBAOC1iS5t/unix_domain_listener'), (b'LOGNAME', b'chembrows'), (b'TERM_PROGRAM', b'Apple_Terminal'), (b'Apple_PubSub_Socket_Render', b'/private/tmp/com.apple.launchd.fzy01S7PKa/Render'), (b'HISTSIZE', b'10000'), (b'SECURITYSESSIONID', b'186dc'), (b'XPC_FLAGS', b'0x0'), (b'USER', b'chembrows'), (b'MANPAGER', b'vimpager'), (b'PWD', b'/Users/chembrows/Desktop/myprogram/dist/myprogram-0.9.1.macosx-10_6-intel/myprogram.app/Contents/MacOS'), (b'SHLVL', b'1'), (b'HISTTIMEFORMAT', b'[ %d/%m/%Y %H:%M:%S ]'), (b'SSH_AUTH_SOCK', b'/private/tmp/com.apple.launchd.Vp8BNWInk0/Listeners'), (b'EDITOR', b'/usr/bin/vim'), (b'RESOURCEPATH', b'/Users/chembrows/Desktop/myprogram/dist/myprogram-0.9.1.macosx-10_6-intel/myprogram.app/Contents/Resources'), (b'__CF_USER_TEXT_ENCODING', b'0x1F7:0x0:0x1'), (b'OLDPWD', b'/Users/chembrows/Desktop/myprogram'), (b'TERM_SESSION_ID', b'20B3A39C-D95D-437E-9694-6CB85BA27C49'), (b'TMPDIR', b'/var/folders/sv/3vhc3x596wq8vfyhlx21qz9m0000gq/T/'), (b'HISTCONTROL', b'ignoreboth'), (b'PYTHONPATH', b'/Users/chembrows/Desktop/myprogram/dist/myprogram-0.9.1.macosx-10_6-intel/myprogram.app/Contents/Resources'), (b'PYTHONDONTWRITEBYTECODE', b'1'), (b'EXECUTABLEPATH', b'/Users/chembrows/Desktop/myprogram/dist/myprogram-0.9.1.macosx-10_6-intel/myprogram.app/Contents/MacOS/./gui'), (b'LC_CTYPE', b'en_US.UTF-8')]
  File "/Users/chembrows/Desktop/myprogram/dist/myprogram-0.9.1.macosx-10_6-intel/myprogram.app/Contents/Resources/__boot__.py", line 704, in <module>
    bootstrap()
  File "/Users/chembrows/Desktop/myprogram/dist/myprogram-0.9.1.macosx-10_6-intel/myprogram.app/Contents/Resources/__boot__.py", line 335, in bootstrap
    return chainload(pathjoin(vsdir,best_version))
  File "/Users/chembrows/Desktop/myprogram/dist/myprogram-0.9.1.macosx-10_6-intel/myprogram.app/Contents/Resources/__boot__.py", line 362, in chainload
    _chainload(target_dir)
  File "/Users/chembrows/Desktop/myprogram/dist/myprogram-0.9.1.macosx-10_6-intel/myprogram.app/Contents/Resources/__boot__.py", line 412, in _chainload
    execv(target_exe,[target_exe] + sys.argv[1:])
PermissionError: [Errno 13] Permission denied
2015-12-03 22:36:21.209 gui[7668:375362] myprogram Error
rfk commented 8 years ago

Interesting, is the matching gui executable in the frozen app marked as executable? Otherwise, I wonder if there's some new security policy in OSX that's blocking this from being executed...

b'EXECUTABLEPATH'

Aha, they values here are bytes rather than strings, that's probably what caused the KeyError.

JPFrancoia commented 8 years ago

Yep, the boostrap exe and the real exe are both marked as executable.

Yes they are bytes. Shall we convert the values before we use them ?

Also, is

/Users/chembrows/Desktop/myprogram/dist/myprogram-0.9.1.macosx-10_6-intel/myprogram.app/Contents/MacOS/./gui

A correct path ? Shouldn't it be

/Users/chembrows/Desktop/myprogram/dist/myprogram-0.9.1.macosx-10_6-intel/myprogram.app/Contents/MacOS/gui
rfk commented 8 years ago

The extra . in the path shouldn't matter IIUC.

Yes they are bytes. Shall we convert the values before we use them

No, I think it's OK to use them as bytes, but the EXTRA_BOOTSTRAP_CODE for py3 probably needs to be updated to use bytes for keys, like:

_EXTRA_BOOTSTRAP_CODE = """
from posix import environ
sys.executable = environ[b"EXECUTABLEPATH"]
...etc...
JPFrancoia commented 8 years ago

I changed the extra_bootstrap_code as you required:

Traceback (most recent call last):
Dec  3 23:03:02  gui[8405] <Notice>:   File "/Users/chembrows/Desktop/myprogram/dist/myprogram-0.9.1.macosx-10_6-intel/myprogram.app/Contents/Resources/__boot__.py", line 704, in <module>
Dec  3 23:03:02  gui[8405] <Notice>:     bootstrap()
Dec  3 23:03:02  gui[8405] <Notice>:   File "/Users/chembrows/Desktop/myprogram/dist/myprogram-0.9.1.macosx-10_6-intel/myprogram.app/Contents/Resources/__boot__.py", line 310, in bootstrap
Dec  3 23:03:02  gui[8405] <Notice>:     sys.executable = abspath(sys.executable)
Dec  3 23:03:02  gui[8405] <Notice>:   File "/Users/chembrows/Desktop/myprogram/dist/myprogram-0.9.1.macosx-10_6-intel/myprogram.app/Contents/Resources/__boot__.py", line 77, in abspath
Dec  3 23:03:02  gui[8405] <Notice>:     path = pathjoin(getcwd(),path)
Dec  3 23:03:02  gui[8405] <Notice>:   File "/Users/chembrows/Desktop/myprogram/dist/myprogram-0.9.1.macosx-10_6-intel/myprogram.app/Contents/Resources/__boot__.py", line 254, in pathjoin
Dec  3 23:03:02  gui[8405] <Notice>:     if isabs(arg):
Dec  3 23:03:02  gui[8405] <Notice>:   File "/Users/chembrows/Desktop/myprogram/dist/myprogram-0.9.1.macosx-10_6-intel/myprogram.app/Contents/Resources/__boot__.py", line 75, in isabs
Dec  3 23:03:02  gui[8405] <Notice>:     return (path.startswith(SEP))
Dec  3 23:03:02  gui[8405] <Notice>: TypeError: startswith first arg must be bytes or a tuple of bytes, not str
rfk commented 8 years ago

Ugh, bytes. @JPFrancoia try this one:

_EXTRA_BOOTSTRAP_CODE = """
from posix import environ
def _decode(b):
    return b.decode(sys.getfilesystemencoding())
sys.executable = _decode(environ[b"EXECUTABLEPATH"])
sys.argv[0] = _decode(environ[b"ARGVZERO"])
"""

Sorry, I haven't managed to find the time to get py2app up and running and make a fresh build myself yet, so I can't try it out myself.

rfk commented 8 years ago

Woohoo, with all of:

I can successfully run the top-level script for the compiled app that you sent me! Hopefully it will also work for you. If so, we can work backwards to figure out what to change in esky itself to make this all work by default.

rfk commented 8 years ago

Paring lib-dynload down to the following still works:

$ ls ./ChemBrows.app/Contents/Resources/lib/python3.5/lib-dynload/
fcntl.so    zlib.so
rfk commented 8 years ago

Paring the contents of python35.zip down to the following still works:

$ unzip ../python35.zip > /dev/null
$ ls
_weakrefset.pyc abc.pyc     codecs.pyc  encodings   io.pyc

AFAICT this is the minimal set of modules required for python3 to start up successfully. So if this works for you @JPFrancoia, the fixes we need to make are:

JPFrancoia commented 8 years ago

WOUHOUHOU :) It also works for me guys.

For now, I did:

as @rfk suggested.

Now we should make everything look good, and everything is in the pocket :)

@timeyyy , I can' find your email address, can you PM it to me ?

rfk commented 8 years ago

@JPFrancoia thanks so much for sticking with this to work through the issue; I think the discoveries here will also help unblock #131 and related issues.

JPFrancoia commented 8 years ago

I'll try to patch Esky with what you guys suggested. @rfk , if I set the content of python35.zip down to the following:

_weakrefset.pyc abc.pyc     codecs.pyc  encodings   io.pyc

I get the 

Fatal Python error: Py_Initialize: unable to load the file system codec
ImportError: No module named 'encodings'

error. Did you do something else to your zip file ? (I also set the the contents of lib-dynload to what you said).

Also, why copying to the bootstrap env ? Wouldn't a link work (something like ln -s) ?

rfk commented 8 years ago

Did you include all the contents of the "encodings" directory in the zipfile as well?

JPFrancoia commented 8 years ago

Yes, here is what I have in the top level Resources folder (I unzipped the python35 archive):

PythonApplet.icns   __boot__.py     __error__.sh        include         lib         site.pyc

./include:
python3.5m

./include/python3.5m:
pyconfig.h

./lib:
python3.5   python35    python35.zip

./lib/python3.5:
lib-dynload site.pyc

./lib/python3.5/lib-dynload:
fcntl.so    zlib.so

./lib/python35:
_weakrefset.pyc abc.pyc     codecs.pyc  encodings   io.pyc

./lib/python35/encodings:
__init__.pyc        cp1255.pyc      cp860.pyc       gb2312.pyc      iso8859_15.pyc      mac_croatian.pyc    shift_jisx0213.pyc
aliases.pyc     cp1256.pyc      cp861.pyc       gbk.pyc         iso8859_16.pyc      mac_cyrillic.pyc    tis_620.pyc
ascii.pyc       cp1257.pyc      cp862.pyc       hex_codec.pyc       iso8859_2.pyc       mac_farsi.pyc       undefined.pyc
base64_codec.pyc    cp1258.pyc      cp863.pyc       hp_roman8.pyc       iso8859_3.pyc       mac_greek.pyc       unicode_escape.pyc
big5.pyc        cp273.pyc       cp864.pyc       hz.pyc          iso8859_4.pyc       mac_iceland.pyc     unicode_internal.pyc
big5hkscs.pyc       cp424.pyc       cp865.pyc       idna.pyc        iso8859_5.pyc       mac_latin2.pyc      utf_16.pyc
bz2_codec.pyc       cp437.pyc       cp866.pyc       iso2022_jp.pyc      iso8859_6.pyc       mac_roman.pyc       utf_16_be.pyc
charmap.pyc     cp500.pyc       cp869.pyc       iso2022_jp_1.pyc    iso8859_7.pyc       mac_romanian.pyc    utf_16_le.pyc
cp037.pyc       cp65001.pyc     cp874.pyc       iso2022_jp_2.pyc    iso8859_8.pyc       mac_turkish.pyc     utf_32.pyc
cp1006.pyc      cp720.pyc       cp875.pyc       iso2022_jp_2004.pyc iso8859_9.pyc       mbcs.pyc        utf_32_be.pyc
cp1026.pyc      cp737.pyc       cp932.pyc       iso2022_jp_3.pyc    johab.pyc       palmos.pyc      utf_32_le.pyc
cp1125.pyc      cp775.pyc       cp949.pyc       iso2022_jp_ext.pyc  koi8_r.pyc      ptcp154.pyc     utf_7.pyc
cp1140.pyc      cp850.pyc       cp950.pyc       iso2022_kr.pyc      koi8_t.pyc      punycode.pyc        utf_8.pyc
cp1250.pyc      cp852.pyc       euc_jis_2004.pyc    iso8859_1.pyc       koi8_u.pyc      quopri_codec.pyc    utf_8_sig.pyc
cp1251.pyc      cp855.pyc       euc_jisx0213.pyc    iso8859_10.pyc      kz1048.pyc      raw_unicode_escape.pyc  uu_codec.pyc
cp1252.pyc      cp856.pyc       euc_jp.pyc      iso8859_11.pyc      latin_1.pyc     rot_13.pyc      zlib_codec.pyc
cp1253.pyc      cp857.pyc       euc_kr.pyc      iso8859_13.pyc      mac_arabic.pyc      shift_jis.pyc
cp1254.pyc      cp858.pyc       gb18030.pyc     iso8859_14.pyc      mac_centeuro.pyc    shift_jis_2004.pyc