Closed mid-kid closed 4 years ago
@frmdstryr That's extremely encouraging, thanks for the research! I'll certainly try this
I was able to get about a 15% reduction by using pyminifier.
Interestingly, it reduces the py files about about 50%, but when compiled to pyo the difference is only ~15%. Still, took the extracted python from 9.6 MB to 8.2 MB and reduced the compressed python by about 0.5MB (3.2MB to 2.7MB). So about 2MB overall, better than nothing.
'''
Copyright (c) 2017, Jairus Martin.
Distributed under the terms of the MIT License.
The full license is in the file COPYING.txt, distributed with this software.
Created on July 21, 2017
@author: jrm
'''
def minifiy_sources(ctx, src):
""" Finds all .py files within a path and runs minification on them.
this should be done BEFORE compileall is run.
Warning: This does IN PLACE minification.
"""
info("Minifying sources, this may take some time...")
from pyminifier import token_utils, minification
class Options(object):
tabs = False #: Replace indentation with tabs (vs single space)
cwd = os.getcwd()
original = 0
minified = 0
passed = 0
total = 0
with current_directory(src):
#: Find all files
for py in shprint(sh.find, '.', '-name', '*.py').split("\n"):
py_file = realpath(join(src, py))
if not exists(py_file) or isdir(py_file):
continue
total +=1
#: Read source code
with open(py_file) as f:
code = f.read()
original += len(code)
try:
#: Minify code
#: The operators seem to have a bug...
minified_code = code
for f in [minification.remove_comments_and_docstrings,
minification.fix_empty_methods,
minification.remove_blank_lines,
minification.reduce_operators,
minification.dedent
]:
minified_code = f(minified_code)
#tokens = token_utils.listified_tokenizer(code)
#minified_code = minification.minify(tokens, Options)
#print("OK",)
except Exception as e:
info("Minifying {}... Failed! Reason: {}".format(py, e))
#raise
continue # Can't minify, leave it as is!
#: Write minified code
with open(py_file, 'wb') as f:
f.write(minified_code)
#: Make sure it compiles
try:
#: Minification compiles to pyo to make sure it's ok
#: Compile pip packages to bytecode
sh.bash('-c',
"source {}/venv/bin/activate && env CC=/bin/false CXX=/bin/false"
"PYTHONPATH={} python -{} -m py_compile {}"
.format(cwd, ctx.get_site_packages_dir(), ctx.optimize, py_file))
minified += len(minified_code)
passed +=1
except Exception as e:
info("Compile {}... Failed! Reason: {}".format(py, e))
#: Undo
with open(py_file, 'wb') as f:
f.write(code)
# #raise
minified += len(code)
#: Optimize original file
sh.bash('-c',
"source {}/venv/bin/activate && env CC=/bin/false CXX=/bin/false"
"PYTHONPATH={} python -{} -m py_compile {}"
.format(cwd, ctx.get_site_packages_dir(), ctx.optimize, py_file))
info_main("Minification complete total={} passed={}({}%) savings={}%!"
.format(total, passed, round(100*passed/total), round(100*minified/original)))
@frmdstryr If you can, try only the minification on the whole Python's Lib
folder recursively. Maybe the best would be not to include it to the recipe, but use only some console arg and do it before actually packaging the application i.e. right before the private.mp3
creation.
I remember trying pyminifier
with pyinstaller
and there it worked just fine even with the obfuscation, however on android we might get into troubles with that. I tried a similar thing with replacing 4 spaces with 1 tab, but the size change wasn't that significant for such a harsh operation and with Kivy itself the change was almost invisible + when you start mixing tabs & spaces in the app, well... :D In any case, minification might even backfire, so if included, we probably shouldn't do it automatically.
@frmdstryr Has anyone researched on Ultimate Executable Packer ? The code is currently way out of my league .. can it be adapted for python pyc/pyo , etc.. ?
Might affect startup performance though.
Every such compression will need decompression on the device. It doesn't really matter how quickly you compress if you end up with the loading screen displayed for a minute. You can't just make the whole interpreter disappear with compressing, you need to remove files from it to actually make a usable difference. Just saying.
@KeyWeeUsr That's what I'm doing, except I'm using crystax python 2. Currently I extract the stdlib.zip
and site-packages
to one folder and run minify on all py files within it recursively so it minifies(?) everything except my app code files. I added a flag to p4a to enable/disable it.
In any case, minification might even backfire, so if included, we probably shouldn't do it automatically.
Yes, exactly. If it does get added, it should be disabled by default as it's more of a final product optimization anyways.
Minify does screw up some files, for instance the sre*.py
modules used by re
in the python Lib
fails to import (even though it compiles to pyo fine?). I also have it skip the join_multiline_pairs
minification as that seemed to cause a lot of issues. But some method of exclusion would also be needed because every user has different requirements.
@kamathln p4a already compresses everything with tar.gz
so there won't be much of a gain there.
It would be interesting if some sort of bundler could be used like what is commonly done with javascript (webpack for example). react-native does this on release builds. It wouldn't save space (unless you can do minification on the bundle) but you could use the trick to have it extracted during install and avoid the startup extraction.
Pros:
Cons:
The only "packer" I found was pinliner. I ran a few tests, and it can't bundle multiple packages and modules at the moment (but does work nice for a single package), and it doesn't pack pyc/pyo files. If these were implemented, the python Lib
and site-packages
could be packed into one big file, then it could be named something that tricks android into copying it with the native modules (ex lib.pymodules.so
) during install and imported from there.
@frmdstryr Thanks for correcting my assumption. Had assumed it is plain tar.
Tried re-compressing the tar.gz with tar.xz instead. My private.mp3 came down from 6.3mb to 4.7mb . The decompression speed was 3x slower than gz though. (still much faster than bz2) . The comparizon was on a laptop though. Planning to try on my phone using termux .
On my phone, in Termux:
$ time bzcat t.tar.bz2 > /dev/null
real 0m3.672s
user 0m3.520s
sys 0m0.030s
$ time zcat t.tgz > /dev/null
real 0m0.489s
user 0m0.420s
sys 0m0.020s
$ time xzcat t.tar.xz > /dev/null
real 0m1.191s
user 0m1.110s
sys 0m0.040s
$
The tar.gz is simply a rename of private.mp3 . tar.xz and tar.bz2 are created with ....
zcat t.tar.gz2 | xz > t.tar.xz
....technique
Can I do it by just deleting blank lines, comments,unused things from kivy???
No, you won't get much if any gain from that. We already compile and ship bytecode and that doesn't help the size much.
Should I use pyminifier?
On Wed, 22 Jul 2020, 1:29 am Alexander Taylor, notifications@github.com wrote:
No, you won't get much if any gain from that. We already compile and ship bytecode and that doesn't help the size much.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kivy/python-for-android/issues/202#issuecomment-662061832, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQGPIPDXL5NJDUETGLZHZVTR4XUAJANCNFSM4ALZVHAA .
the issue is not the size of python code, it's mostly the size of binaries, like python itself, and all the cpython extensions (like kivy, but also others), that can quickly run into the megabyte range.
I'm not sure this issue is actionable, it's too broad, there will always be reasons to argue that apps are "too big", i'd rather see someone analyse and point at specific issues that can be fixed to make the apk / installed size smaller, and to have these fixed independently. So i'm going to close it, if another maintainer disagree, feel free to reopen, if an user sees a specific issue to fix to help address the point of this issue, feel free to open another one describing the exact issue you see.
Can this be achieved by removing unused libraries and packages?
Every separate kivy app installed has an identical copy of the libraries. That makes every kivy app be about 20MB in size when only including the kivy library and it's dependencies. That's very big considering the actual content of the app.
There are several options to solve this problem I can think of:
Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.