Support for Python zipapp module

[leetschau]

zipapp makes multiple Python files as a bundle to be easy distributed. However when the pye file is bundled into the zipapp, it can't be found. Here is a minimal demo:

$ cat add_cols/app.py
import pyconcrete
import sys
from mylib import add2col

def main():
    inp = sys.argv[1]
    out = sys.argv[2]
    print(f'read file {inp}, and convert to file {out}')
    add2col(inp, out)

if __name__ == '__main__':
    main()

$ cat add_cols/mylib.py
import pandas as pd

def add2col(inpfile: str, outfile: str):
    inp = pd.read_csv(inpfile)
    inp['cola'] = inp['colb'] + inp['cola']
    inp.to_csv(outfile, index=False)

$ cat demo.csv
cola,colb
1,2
1,2
1,2
1,2

## run the plain Python file version:
$ python -m zipapp add_cols -m 'app:main'   # create a zipapp with plain Python file
$ python add_cols.pyz demo.csv res.csv   # the result file 'res.csv' is created properly

## replace mylib.py with encrypted version and run again:
$ cd add_cols
$ pyconcrete-admin.py compile --source=mylib.py --pye
$ rm mylib.py
$ cd ..
$ python -m zipapp add_cols -m 'app:main'   # create a encrypted version zipapp
$ python add_cols.pyz demo.csv res.csv       # run again
Traceback (most recent call last):
  File "/opt/app/anaconda3/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/app/anaconda3/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "add_cols.pyz/__main__.py", line 2, in <module>
  File "add_cols.pyz/app.py", line 3, in <module>
ModuleNotFoundError: No module named 'mylib'

Environment:

• Python 3.6
• Ubuntu 16.04

I have no experience in C/C++, and can not make more debuggings.

Thanks for help.

[Falldog]

Do you open the .pyz file? Does it contain mylib.pye?

[leetschau]

@Falldog Yes, mylib.pye is in add_cols.pyz, which is just a plain zip file:

$ unzip -l add_cols.pyz
Archive:  add_cols.pyz
  Length      Date    Time    Name
---------  ---------- -----   ----
      352  01-21-2019 16:19   mylib.pye
      231  01-21-2019 16:20   app.py
       46  01-23-2019 15:00   __main__.py
---------                     -------
      629                     3 files

[coldfire0200]

I have a trivial implementation that allows pye files in a zip package. I tested with python 3.10 on Windows 11, the zip package contains 100+ pye files in complex directory structure and everything works just like pyc files (a bit slower though). The package has to end with .zip extension (as the regex use it to parse the file path) but you can make changes to support other format. My fork: https://github.com/coldfire0200/pyconcrete/tree/master/src/pyconcrete

Basically in the original code, exists function is used to check file existence and open function is used to open file. This would work with regular file system but not files within a zip package. In my implementation when the file path contains .zip file the check existence and file read are handled differently (using the ZipFile operations).

I do agree with other discussion that at least PyeLoader should be built as binary (Cython?) to make it a little bit more difficult to get the decrypted pyc file.

[Falldog]

@coldfire0200 awesome work! PullRequest is welcome, if it could contain more test-cases would be better If you just left fork here, I will spent some time to add the test-case and merge it later, maybe it will take time, lol.

I do agree with other discussion that at least PyeLoader should be built as binary (Cython?) to make it a little bit more difficult to get the decrypted pyc file.

Agree, it would be a new topic for next steps

[coldfire0200]

@coldfire0200 awesome work! PullRequest is welcome, if it could contain more test-cases would be better If you just left fork here, I will spent some time to add the test-case and merge it later, maybe it will take time, lol.

I do agree with other discussion that at least PyeLoader should be built as binary (Cython?) to make it a little bit more difficult to get the decrypted pyc file.

Agree, it would be a new topic for next steps

Thanks I will spend some time writing test cases this weekend.