Apps are too big (in size)

mid-kid commented 10 years ago

Every separate kivy app installed has an identical copy of the libraries. That makes every kivy app be about 20MB in size when only including the kivy library and it's dependencies. That's very big considering the actual content of the app.

There are several options to solve this problem I can think of:

Go the necessitas/sl4a way and have an app for the libraries only. Users will be annoyed.
Have every app include a piece of code to download precompiled libs (+ deps) to a .kivy folder in the sdcard. The sdcard needs to be available and have enough space.
Use tinypy (though I doubt the compatibility + the project is pretty much moribund)
Instead of extracting the python libraries from a zip, I think the python interpreter can load the zip directly. This will help a bit for size, but not for (startup) performance.
I think nuitka can include all the necessary scripts into a single binary, then you just need several .so files (libpython, sdl, etc) and in my crappy theory, that should work. I don't know if this will reduce the size much, but at least you get performance, so that's a plus, I guess.
Any ideas?

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

inclement commented 7 years ago

@frmdstryr That's extremely encouraging, thanks for the research! I'll certainly try this

frmdstryr commented 7 years ago

I was able to get about a 15% reduction by using pyminifier.

Interestingly, it reduces the py files about about 50%, but when compiled to pyo the difference is only ~15%. Still, took the extracted python from 9.6 MB to 8.2 MB and reduced the compressed python by about 0.5MB (3.2MB to 2.7MB). So about 2MB overall, better than nothing.

'''
Copyright (c) 2017, Jairus Martin.

Distributed under the terms of the MIT License.

The full license is in the file COPYING.txt, distributed with this software.

Created on July 21, 2017

@author: jrm
'''

def minifiy_sources(ctx, src):
    """ Finds all .py files within a path and runs minification on them.
        this should be done BEFORE compileall is run.

        Warning: This does IN PLACE minification.
    """

    info("Minifying sources, this may take some time...")
    from pyminifier import token_utils, minification

    class Options(object):
        tabs = False  #: Replace indentation with tabs (vs single space)

    cwd = os.getcwd()
    original = 0
    minified = 0
    passed = 0
    total = 0
    with current_directory(src):
        #: Find all files
        for py in shprint(sh.find, '.', '-name', '*.py').split("\n"):
            py_file = realpath(join(src, py))
            if not exists(py_file) or isdir(py_file):
                continue
            total +=1
            #: Read source code
            with open(py_file) as f:
                code = f.read()
                original += len(code)
            try:
                #: Minify code
                #: The operators seem to have a bug...
                minified_code = code
                for f in [minification.remove_comments_and_docstrings,
                          minification.fix_empty_methods,
                          minification.remove_blank_lines,
                          minification.reduce_operators,
                          minification.dedent
                          ]:
                    minified_code = f(minified_code)

                #tokens = token_utils.listified_tokenizer(code)
                #minified_code = minification.minify(tokens, Options)
                #print("OK",)
            except Exception as e:
                info("Minifying {}... Failed! Reason: {}".format(py, e))
                #raise
                continue  # Can't minify, leave it as is!

            #: Write minified code
            with open(py_file, 'wb') as f:
                f.write(minified_code)

            #: Make sure it compiles
            try:
                #: Minification compiles to pyo to make sure it's ok
                #: Compile pip packages to bytecode
                sh.bash('-c',
                        "source {}/venv/bin/activate && env CC=/bin/false CXX=/bin/false"
                        "PYTHONPATH={} python -{} -m py_compile {}"
                        .format(cwd, ctx.get_site_packages_dir(), ctx.optimize, py_file))

                minified += len(minified_code)
                passed +=1
            except Exception as e:
                info("Compile {}... Failed! Reason: {}".format(py, e))
                #: Undo
                with open(py_file, 'wb') as f:
                    f.write(code)
            #     #raise
                minified += len(code)

                #: Optimize original file
                sh.bash('-c',
                        "source {}/venv/bin/activate && env CC=/bin/false CXX=/bin/false"
                        "PYTHONPATH={} python -{} -m py_compile {}"
                        .format(cwd, ctx.get_site_packages_dir(), ctx.optimize, py_file))

    info_main("Minification complete total={} passed={}({}%) savings={}%!"
         .format(total, passed, round(100*passed/total), round(100*minified/original)))

KeyWeeUsr commented 7 years ago

@frmdstryr If you can, try only the minification on the whole Python's Lib folder recursively. Maybe the best would be not to include it to the recipe, but use only some console arg and do it before actually packaging the application i.e. right before the private.mp3 creation.

I remember trying pyminifier with pyinstaller and there it worked just fine even with the obfuscation, however on android we might get into troubles with that. I tried a similar thing with replacing 4 spaces with 1 tab, but the size change wasn't that significant for such a harsh operation and with Kivy itself the change was almost invisible + when you start mixing tabs & spaces in the app, well... :D In any case, minification might even backfire, so if included, we probably shouldn't do it automatically.

kamathln commented 7 years ago

@frmdstryr Has anyone researched on Ultimate Executable Packer ? The code is currently way out of my league .. can it be adapted for python pyc/pyo , etc.. ?

Might affect startup performance though.

KeyWeeUsr commented 7 years ago

Every such compression will need decompression on the device. It doesn't really matter how quickly you compress if you end up with the loading screen displayed for a minute. You can't just make the whole interpreter disappear with compressing, you need to remove files from it to actually make a usable difference. Just saying.

frmdstryr commented 7 years ago

@KeyWeeUsr That's what I'm doing, except I'm using crystax python 2. Currently I extract the stdlib.zip and site-packages to one folder and run minify on all py files within it recursively so it minifies(?) everything except my app code files. I added a flag to p4a to enable/disable it.

In any case, minification might even backfire, so if included, we probably shouldn't do it automatically.

Yes, exactly. If it does get added, it should be disabled by default as it's more of a final product optimization anyways.

Minify does screw up some files, for instance the sre*.py modules used by re in the python Lib fails to import (even though it compiles to pyo fine?). I also have it skip the join_multiline_pairs minification as that seemed to cause a lot of issues. But some method of exclusion would also be needed because every user has different requirements.

@kamathln p4a already compresses everything with tar.gz so there won't be much of a gain there.

frmdstryr commented 7 years ago

It would be interesting if some sort of bundler could be used like what is commonly done with javascript (webpack for example). react-native does this on release builds. It wouldn't save space (unless you can do minification on the bundle) but you could use the trick to have it extracted during install and avoid the startup extraction.

Pros:

Importing would be one file, which "could" potentially save a lot of system calls and speedup importing (theoretically, I have no data to back it up). Currently python itself is ~700 files which is a lot of blocking system calls.
It could be extracted on install by tricking android into thinking it's a native lib, saving extraction time on first startup.

Cons:

Would still need to extract any data files.
Would need copied for each arch (if multiple are used)

The only "packer" I found was pinliner. I ran a few tests, and it can't bundle multiple packages and modules at the moment (but does work nice for a single package), and it doesn't pack pyc/pyo files. If these were implemented, the python Lib and site-packages could be packed into one big file, then it could be named something that tricks android into copying it with the native modules (ex lib.pymodules.so) during install and imported from there.

kamathln commented 7 years ago

@frmdstryr Thanks for correcting my assumption. Had assumed it is plain tar.

Tried re-compressing the tar.gz with tar.xz instead. My private.mp3 came down from 6.3mb to 4.7mb . The decompression speed was 3x slower than gz though. (still much faster than bz2) . The comparizon was on a laptop though. Planning to try on my phone using termux .

kamathln commented 7 years ago

On my phone, in Termux:

$ time bzcat t.tar.bz2 > /dev/null

real    0m3.672s
user    0m3.520s
sys     0m0.030s
$ time zcat t.tgz > /dev/null

real    0m0.489s
user    0m0.420s
sys     0m0.020s

$ time xzcat t.tar.xz > /dev/null

real    0m1.191s
user    0m1.110s
sys     0m0.040s
$

kamathln commented 7 years ago

The tar.gz is simply a rename of private.mp3 . tar.xz and tar.bz2 are created with ....

zcat t.tar.gz2 | xz > t.tar.xz

....technique

tasen-sp commented 4 years ago

Can I do it by just deleting blank lines, comments,unused things from kivy???

inclement commented 4 years ago

No, you won't get much if any gain from that. We already compile and ship bytecode and that doesn't help the size much.

tasen-sp commented 4 years ago

Should I use pyminifier?

On Wed, 22 Jul 2020, 1:29 am Alexander Taylor, notifications@github.com wrote:

No, you won't get much if any gain from that. We already compile and ship bytecode and that doesn't help the size much.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/kivy/python-for-android/issues/202#issuecomment-662061832, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQGPIPDXL5NJDUETGLZHZVTR4XUAJANCNFSM4ALZVHAA .

tshirtman commented 4 years ago

the issue is not the size of python code, it's mostly the size of binaries, like python itself, and all the cpython extensions (like kivy, but also others), that can quickly run into the megabyte range.

I'm not sure this issue is actionable, it's too broad, there will always be reasons to argue that apps are "too big", i'd rather see someone analyse and point at specific issues that can be fixed to make the apk / installed size smaller, and to have these fixed independently. So i'm going to close it, if another maintainer disagree, feel free to reopen, if an user sees a specific issue to fix to help address the point of this issue, feel free to open another one describing the exact issue you see.

omeco4christ commented 2 years ago

Can this be achieved by removing unused libraries and packages?

kivy / python-for-android

Apps are too big (in size) #202