kivy / python-for-android

Turn your Python application into an Android APK
https://python-for-android.readthedocs.io
MIT License
8.33k stars 1.85k forks source link

Apps are too big (in size) #202

Closed mid-kid closed 4 years ago

mid-kid commented 10 years ago

Every separate kivy app installed has an identical copy of the libraries. That makes every kivy app be about 20MB in size when only including the kivy library and it's dependencies. That's very big considering the actual content of the app.

There are several options to solve this problem I can think of:

Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.

brousch commented 10 years ago

We rehash this problem every few months. As you've discovered, there is no ideal solution. On Feb 1, 2014 5:59 PM, "mid-kid" notifications@github.com wrote:

Every separate kivy app installed has an identical copy of the libraries. That makes every kivy app be about 20MB in size when only including the kivy library and it's dependencies. That's very big considering the actual content of the app.

There are several options to solve this problem I can think of:

  • Go the necessitas way and have an app for the libraries only. Users will be annoyed.
  • Have every app include a piece of code to download precompiled libs (+ deps) to a .kivy folder in the sdcard. The sdcard needs to be available.
  • Use tinypy (though I doubt the compatibility + the project is pretty much morubund)
  • Instead of extracting the python libraries from a zip, I think the python interpreter can load the zip directly. This will help a bit for size, but not for (startup) performance.
  • I think nuitka can include all the necessary scripts into a single binary, then you just need several .so files (libpython, sdl, etc) and in my crappy theory, that should work. I don't know if this will reduce the size much, but at least you get performance, so that's a plus, I guess.
  • Any ideas?

Reply to this email directly or view it on GitHubhttps://github.com/kivy/python-for-android/issues/202 .

akshayaurora commented 10 years ago

Every kivy app is not 20MB, look at the showcase. It's 6MB, you need to trim your app. Kivy itself comes under 5-6 mb as evident by the showcase.

tito commented 10 years ago

And more, but all the things we currently explored have much more drawbacks than usability. The current state is a mix. It can always be improved, but must not be complex to use as well.

tito commented 10 years ago

And yes, a minimal APK size is ~5MB. 20MB is already big. And most of the time, it's due to the data in your application, not Kivy.

brousch commented 10 years ago

It is 20MB after extraction. If you ;look in Settings->Apps you can dee this size. Showcase is about 35MB. On Feb 1, 2014 6:18 PM, "Mathieu Virbel" notifications@github.com wrote:

And yes, a minimal APK size is ~5MB. 20MB is already big. And most of the time, it's due to the data in your application, not Kivy.

Reply to this email directly or view it on GitHubhttps://github.com/kivy/python-for-android/issues/202#issuecomment-33886483 .

mid-kid commented 10 years ago

I see I missed a point there: I was pretty much talking about the size in the internal memory after extraction and moved to SD, as shown by the android system package manager in the downloaded apps tab, which may be a bit problematic on devices low on internal memory. The app I got the numbers with is a self-compiled launcher with only kivy (master branch) + deps.

Please note that the package manager has calculated the size wrong since as far as I can remember. Just try to move an app to/from the SD and look at the total.

Sorry for the confusion.

adamski commented 10 years ago

This is my main concern before delving into learning Kivy for mobile app development... I'm currently using JUCE and C++ but would rather be using Python for main part of the app.. however, app size and battery drain are concerns that I've seen raised about Kivy mobile apps.

tshirtman commented 10 years ago

I do think 6-7Megs are acceptable sizes for an app these days, and a lot more if you have data, you rarely ever see less than one meg apps anymore, imho. For battery drain, i never witnessed anything about kivy that would cause that, i do have a few unity apps that makes my phone burn my hand and empty my battery pretty quick (nexus 4), nothing of that kind happened with any kivy app that i tested.

mid-kid commented 10 years ago

@tshirtman

It may not be much of a problem on some newer phones with more storage, but if you can make it a bit smaller, and in the process support more phones, why not? It can clearly be optimized. And yes, <1m apps are rare, but I'm not talking about that. There's a big difference between that, and 20M (which makes it to the top 10 biggest apps on my phone).

P.S. I'm not talking about performance.

brousch commented 10 years ago

"Why not?" If we could do it without a bunch of extra work we would. This has not been solved, so it has not been done. On Jul 20, 2014 7:34 AM, "mid-kid" notifications@github.com wrote:

@tshirtman https://github.com/tshirtman

It may not be much of a problem on some newer phones with more storage, but if you can make it a bit smaller, and in the process support more phones, why not? It can clearly be optimized. And yes, <1m apps are rare, but I'm not talking about that. There's a big difference between that, and 20M (which makes it to the top 10 biggest apps on my phone).

P.S. I'm not talking about performance.

— Reply to this email directly or view it on GitHub https://github.com/kivy/python-for-android/issues/202#issuecomment-49544149 .

mid-kid commented 10 years ago

Please note the "if you can" ;)

adamski commented 10 years ago

I'm seeing two sizes stated here - ±20mb and 6-7mb - is the smaller size the compressed installer one would download from the app store? For my needs (musical tuning app), comparable apps are around 500kb on the app store.. so in this case, Kivy is not for me, but I am considering it for building API consuming mobile apps at the company I work for, as in this case size should not be such an issue.

brousch commented 10 years ago

6MB is the smallest size for the APK. Once installed, it will be reported as about 20MB in Android App Settings. On Jul 20, 2014 10:22 AM, "Adam Wilson" notifications@github.com wrote:

I'm seeing two sizes stated here - ±20mb and 6-7mb - is the smaller size the compressed installer one would download from the app store? For my needs (musical tuning app), comparable apps are around 500kb on the app store.. so in this case, Kivy is not for me, but I am considering it for building API consuming mobile apps at the company I work for, as in this case size should not be such an issue.

— Reply to this email directly or view it on GitHub https://github.com/kivy/python-for-android/issues/202#issuecomment-49547667 .

mid-kid commented 10 years ago

A kivy APK is 6MB, once it is run for the first time, libraries are extracted, which ends up as 20MB (Compressed data + estracted data). I can confirm that the android app manager is not the only who reports so, diskusage does too.

akshayaurora commented 10 years ago

@mid-kid. This is a known limitation. Kivy might not be suited for every purpose out there. there have been some attempts at removing some more default modules of python that are usually not needed like distuils etc.

Expecting the kivy apk to be around 500KB though might not be feasible, yet!

We need a good selection mechanism they while packaging figures out all the libs needed and only includes those files that are actually being used.

On Sun, Jul 20, 2014 at 8:38 PM, mid-kid notifications@github.com wrote:

A kivy APK is 6MB, once it is run for the first time, libraries are extracted, which ends up as 20MB (Compressed data + estracted data). I can confirm that the android app manager is not the only who reports so, diskusage does too.

— Reply to this email directly or view it on GitHub https://github.com/kivy/python-for-android/issues/202#issuecomment-49549502 .

knappador commented 10 years ago

Can we delete the zip after extraction?

On Mon, Jul 21, 2014 at 1:02 AM, Akshay Arora notifications@github.com wrote:

@mid-kid. This is a known limitation. Kivy might not be suited for every purpose out there. there have been some attempts at removing some more default modules of python that are usually not needed like distuils etc.

Expecting the kivy apk to be around 500KB though might not be feasible, yet!

We need a good selection mechanism they while packaging figures out all the libs needed and only includes those files that are actually being used.

On Sun, Jul 20, 2014 at 8:38 PM, mid-kid notifications@github.com wrote:

A kivy APK is 6MB, once it is run for the first time, libraries are extracted, which ends up as 20MB (Compressed data + estracted data). I can confirm that the android app manager is not the only who reports so, diskusage does too.

— Reply to this email directly or view it on GitHub < https://github.com/kivy/python-for-android/issues/202#issuecomment-49549502>

.

— Reply to this email directly or view it on GitHub https://github.com/kivy/python-for-android/issues/202#issuecomment-49550908 .

mid-kid commented 10 years ago

What is it with people thinking I expect it to be under 1M? xD Anyway, yes, @akshayaurora, that is why I opened this issue. And I am too lazy to try myself, that's why I'm not expecting anything.

@knappador, no, the zip is contained inside the APK, and as such, it can't be removed.

We can try to remove some unneeded libs, but I don't think that would get very far. Some stripping could do something, like @tito said (I haven't looked at this project in some time now, so I don't know about the state of affairs), but I think the key resides in changing the distribution method of the libraries. No option proposed so far is ideal (imo neither is the current one).

Funny how much activity after 6 months.

knappador commented 10 years ago

The possibility of not including everything in the .mp3 files is interesting and would be more effective than stripping ever could be. The entire tar/mp3 trick was done specifically to deal with the Android toolchain deleting files that we wanted to include in the APK. Maybe we can switch to only packaging files that the Android toolchain mistakenly deletes.

I'm currently looking into a data persistence issue when using --private. I bet the two problems cover the same ground.

hamedmohammadi commented 10 years ago

what aboat to use nuitka and compile all application into c and then use directly into ndk ? it has some optimazation like pack all used package(in my reading in the site)

mid-kid commented 10 years ago

If you read the thread, that question is already answered. Yes it is possible, yes it will speed it up a bit, no it won't reduce the size, no it ain't easy (probably a lot of compile errors). El dia 07/09/2014 10.36, hamed mohammadi notifications@github.com va escriure:what aboat to use nuitka and compile all application into c and then use directly into ndk ? it has some optimazation like pack all used package(in my reading in the site)

—Reply to this email directly or view it on GitHub.

tito commented 10 years ago

Lot of activity here. One more addition to the thread:

I'm currently experimenting a new way to boot Python application (not related to Kivy yet): bootstrap. The minimal bootstrap is based on NDK, extract only one file, setup the path to include zip import, have a mechanism to load CPython extensions library from the libs/ directory (so unpacked during the installation). Boot to python is really faster. BUT i wonder how far we can push this approach with all the python library.

the .egg format separate the libraries from the python code. This is interesting to use if we want to separate thoses in the final apk. However, the real issue is the data shipped with any python library. If you take Kivy, it will search for the glsl shaders in its own data/glsl directory. But it except a posix path, not a path within a zip. Same for images. Now try to think about a method that could work for any python packages: i didn't find one yet.

hamedmohammadi commented 9 years ago

tito your last answer was Great!!! may be it give some hope to use embedding python in widder situations. thanks a lot

hamedmohammadi commented 9 years ago

i dont know it can be helpfull but if there is a way to use jython in android we can add it as a jar file and we reduce the size of python itself

mid-kid commented 9 years ago

Jython does not support python3, and afaik, it does not work with native libraries compiled for CPython, such as SDL.

endolith commented 9 years ago

Kivy sounds perfect, being Python and cross-platform, but then all the apps it creates are huge for what they do, which makes me feel like it's not worth learning.

Every kivy app is not 20MB, look at the showcase. It's 6MB, you need to trim your app. Kivy itself comes under 5-6 mb as evident by the showcase.

On Android it's 37 MB:

kivy showcase 37 mb

How do you "trim your app"? Is this not automatic?

inclement commented 8 years ago

@endolith: @akshayaurora was referring to the .apk file size (where 6-7MB is the minimum), not the unpacked size (where the ~30MB you see is a normal unpacked size for that minimal APK).

By trimming the app, I think @akshayaurora means removing parts of the python installation (and potentially our own app) that aren't necessary. Some such things are done automatically, where modules are unlikely to ever be useful on android, but doing it on a large scale would be extremely difficult as it's hard to inspect a python app and work out what modules it will need. Even doing this manually, I suspect it would be difficult to cut more than 1-2MB from the APK (though that's more difference when extracted).

J-qak commented 8 years ago

One utopic solution to this would be to have Python+Kivy commonly preinstaled on Android systems :-3

v3ss0n commented 8 years ago

make an android virus that automatically spreads and install Kivy , Cordova + crosswalk , etc :P

mardon86 commented 8 years ago

Is there a way to create apk that contain only python script and additional module (without including python vm + kivy) ? and then have the apk detect if the python+kivy is installed. if python+kivy is not installed then download the python+kivy. Therefore every kivy app apk size will not too big.

inclement commented 8 years ago

You could certainly do this. We would probably accept a PR adding such functionality (in a reasonable option way), but are unlikely to work on it any time soon because of the disadvantages this approach still has.

On 10/03/16 08:05, Maradona wrote:

Is there a way to create apk that contain only python script and additional module (without including python vm + kivy) ? and then have the apk detect if the python+kivy is installed. if python+kivy is not installed then download the python+kivy. Therefore every kivy app apk size will not too big.

— Reply to this email directly or view it on GitHub https://github.com/kivy/python-for-android/issues/202#issuecomment-194723497.

admincoder commented 8 years ago

What about compiling the entire application with Nuitka and/or Cython and let the C/C++ compilers perform Dead Code Stripping/Elimination? @inclement, @tito This should strip out all unused parts (unreachable code) of Kivy, SDL2, Python, your own App and any external libraries. I.e As Small As Possible (except for the code size increases made by Nuitka or Cython) 😄

inclement commented 8 years ago

I think that Cython will mostly just make things bigger - I don't have any numbers to hand, but I'm pretty sure this has been discussed before. @tito probably has a stronger idea about it. For instance, Kivy is already heavily cythonised, this doesn't make it smaller in the APK.

I haven't looked at nuitka much, but I'd be surprised if the same weren't true.

In either case, I believe about 4MB of the 7MB basic APK size comes from the Python interpreter and other dependencies (now SDL2 etc., previously Pygame).

admincoder commented 8 years ago

@inclement, you are right that cython makes the files bigger, especially when building for multiple architectures. I'm pretty sure nuitka also enlarges the files (not as much as cython though).

But the key idea here would be that the C/C++ compilers (Clang, GCC, MinGW, Visual C++ etc.) have Optimization options which include "Dead Code Elimination" (-o2 for GCC). This will remove unreachable code (unused symbols) from your entire project which includes Kivy and all dependencies.

Something so good always has a catch.

And that catch is that the entire project must be in C/C++ with nothing remaining in Python, as the interpreter will also be stripped of any unused portions of the VM (from the perspective of the Compiler) rendering parts of python code foreign to the interpreter. With that in mind, this should only be done at the end of the development cycle just before deployment and not before.

mcroni commented 8 years ago

another way you can reduce the size of kivy apps is by editing the file called blacklist.txt this file includes all the modules or dependencies in kivy and you can blacklist them to exclude them(libraries) and this will make your kivy file smaller in size when compiling

mcroni commented 8 years ago

blacklist.txt thats an example of a blacklist.txt file

admincoder commented 8 years ago

@mcroni, Blacklist.txt is quite a handy way to exclude any unused modules, if that is good enough for you.

But passing an entire C/C++ project with a compiler optimization option such as https://en.m.wikipedia.org/wiki/Dead_code_elimination, will work with more granularity than a human would ever want to: It removes unused variables, functions and classes from even every module you do decide to include (even parts of the python interpreter).

This won't be possible by editing the blacklist.txt file, and if it works as expected, YOU WILL NEVER HAVE TO WORRY ABOUT DEPENDENCIES.

Pretty cool right...(Don't take the capitalization personally, just wanted to emphasize that point, as dependency management can be a nightmare when trying to optimize for size).

KeyWeeUsr commented 8 years ago

@admincoder This is all in a theoretical way. Try to make it this way and show some numbers here. If there's no rapid change or even increased size with cythonizing everything(which should), then it doesn't really matter. That's why cython is used in the cases where it should be used(speed) and not where it could be used e.g. even creating a widget.

mcroni commented 8 years ago

well @admincoder i experimented with this a bit and make some mistakes but now i know which modules not to touch, i now have a customized blacklist for my builds. it doesnt hurt at all

admincoder commented 8 years ago

@keyweeusr, I'm sorry, I have a tendency to think about solutions without actually implementing them; due to a lack of experience and time. Please refer to my previous post where I explicitly say Nuitka and/or Cython: to be more clear, Cython for speed ONLY, and Nuitka for compatibility and a pure C/C++ code base should be the ideal steps to achieve what I am talking about. You may even use Nuitka alone, apart from the cythonized modules Kivy provides.

@mcroni, If it doesn't hurt, it doesn't hurt. But I think this post is for people it hurts. IMHO, Kivy has the potential to target smart watches and embedded boards, not to mention other platforms. And also, disk space on smartphones isn't the same as desktops; users may decide to choose another app, delete your app later, or not download it at all simply because of it's size (I've done all three myself).

cdb1019 commented 7 years ago

What is the latest on this issue? Have any improvements been made? Is Kivy still under active development?

inclement commented 7 years ago

Kivy and python-for-android are both actively developed (you can check the recent commits in either of them), but the technical barriers described earlier in this thread have not really changed.

On 28/03/17 03:45, Chris Burchett wrote:

What is the latest on this issue? Have any improvements been made? Is Kivy still under active development?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kivy/python-for-android/issues/202#issuecomment-289646458, or mute the thread https://github.com/notifications/unsubscribe-auth/ABNQm4lkaWD9u70CpdQBP3vR0cHFkkaIks5rqHRTgaJpZM4BeanA.

KeyWeeUsr commented 7 years ago

@cdb1019 I have a small list of files&folders that could be removed from the Python packaged into APK archive, but a testing is needed:

folders:

files:

maybe:

That kills another ~1.86MB from unpacked Python (~7.5MB for me). The rest is only about cutting off stuff from Kivy that you don't need such as behaviors, complex widgets, kivy logos, keyboard layouts, fonts, effects, extras, recorder(input), some modules, storage (if you don't use it), tools (aren't packaged, I think). For all that you'll need to check if they aren't imported or referenced somewhere else (and that's a matter of change always). We can't really strip the Kivy parts as we don't have a clue what a user might use, but we might be able to strip Python a little bit more in the future.

kamathln commented 7 years ago

Anyone has plans to make an Android fork/mod/ROM with Kivy built in? - In such a way that all App developers need to do is package the main.py and main.kivy (obfuscated if necessary), and necessary support files not included with kivy. The install file also comes with a requirements.txt which the internal pip interprets.

Just a thought that crossed my mind. I don't see it growing big without going the way of Firefox OS. It is just for fun.

mcroni commented 7 years ago

I think what you're saying is too vague,, are you referring to something like a kivy launcher but can also compile the apk at the same time??

On Monday, April 10, 2017, Laxminarayan G Kamath A notifications@github.com wrote:

Anyone has plans to make an Android fork/mod/ROM with Kivy built in? - In such a way that all App developers need to do is package the main.py and main.kivy (obfuscated if necessary), and necessary support files not included with kivy. The install file also comes with a requirements.txt which the internal pip interprets.

Just a thought that crossed my mind.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kivy/python-for-android/issues/202#issuecomment-292888833, or mute the thread https://github.com/notifications/unsubscribe-auth/AJGdRDhZcfsyqp9Ec-G1hK0uGsxmTqQdks5rue4agaJpZM4BeanA .

kamathln commented 7 years ago

Yes, It is purposefully vague at the moment. It is more of a problem statement than the solution. The final aim is to be able to distribute APKs or may be some other installer without the whole load of the Interpreter + libraries in every APK/installer. Also, the required python libraries can be installed centrally to reduce duplication.

But on giving it proper thought, I realized the duplication problem can't be correctly solved , as upgrading a library for one app would probably break other apps.

kamathln commented 7 years ago

Now this comment has nothing to do with the above idea. Have we tried experimenting with xz/lzma compression instead of zip? I know, when the app is decompressed initially it is finally going to occupy the same space, but at least it can reduce the compressed size?

frmdstryr commented 7 years ago

Perhaps a modified version of cx_Freeze could be used?

With cx_Freeze, you tell it which modules are needed for your app (already in the kivy requirements), then it scans all those modules and finds their dependencies (you can add ones it can't find automatically) and packs ONLY the needed python modules and their dependencies into a single library.zip file. Also it renames all .so files (only from required modules / packages) in the format package.name.so and places them in a single folder.

build/library.zip # all data, py/pyc files
build/libx.so
build/libpython.so
build/my.pkg.so
etc...

Once this is done:

  1. The library.zip could be extracted and packaged in the apk (unzipped). Since the apk is a zip, if the apk were added to the python path, the python files could then be imported directly with the standard zipimporter (maybe would need a modified path).
  2. All the necessary .so files could be copied to the lib folder so they are extracted upon install.

Pros:

  1. Automatically remove all unused modules (thus reducing overall footprint, i've seen 3x-4x reductions on desktop apps)
  2. Save space of unpacked python assets (current cost is tar.gz size + unpacked size) as they would be loaded right from the apk (excluding .so files)
  3. Remove startup time cost of unpacking assets on first run (since this would be done at install)

Cons:

  1. Would need tested and implemented
  2. Adds a startup loading time due to importing from zip (Importing only the stdlib.zip from crystax only takes 89ms to load)
  3. Loader would need to know to import so files based on dotted name instead of path.

Edit: I tested this by packing everything into a zip, putting the so files in a single location, hacking the import to load the correct so files, and the startup time was rather slow (extra 4 seconds) due to importing from the zipfile. So some alternative way needs investigated.

Perhaps extraction could be sped up using czipfile, or the library contents could be loaded into memfs or something.

frmdstryr commented 7 years ago

I found out how to have all the .so files copied during app install (not on first run!) and get loaded properly from the app.

Doing it this way you can support multiple arch's in one apk, EVEN when your python requirements has compiled components as only the actual py/pyc/pyo files are in the assets folder. You can also then compress all the python libs into a zip and import them from the zip (although it's slower at the moment) or extract as normal.

  1. You have to move all .so files to your libs/<arch>/ folder (where libs is the jniDir)
  2. Rename them to lib.<pkg>.<of>.module.so (if they dont start with lib, android won't copy them from the apk automatically)
  3. Add the module importer to your app before importing anything using the modules and add it with sys.meta_path.append(AndroidFinder())
  4. Add the APK_LIB_DIR to your environment, can be set via the jni and retrieved using getApplicationInfo().nativeLibraryDir in your activity.

'''
Copyright (c) 2017, Jairus Martin.

Distributed under the terms of the MIT License.

The full license is in the file COPYING.txt, distributed with this software.

Created on July 10, 2017

@author: jrm
'''
import os
import sys
import imp
from glob import glob

class AndroidLoader(object):
    """ Loads renamed .so files from the app's lib folder"""

    def load_module(self, mod):
        try:
            return sys.modules[mod]
        except KeyError:
            pass

        lib = AndroidFinder.so_modules[mod]
        m = imp.load_dynamic(mod, lib)
        sys.modules[mod] = m
        return m

class AndroidFinder(object):
    """ Loads renamed so files from the app's lib folder"""
    so_modules = {}

    def __init__(self):
        #: Find all included so files
        lib_dir = os.environ['APK_LIB_DIR']

        for lib in glob('%s/lib.*.so'%lib_dir):
            name = lib.split("/")[-1] # Lib filename
            mod = ".".join(name.split(".")[1:-1])  # Strip lib and so
            AndroidFinder.so_modules[mod] = lib

    def find_module(self, mod, path = None):
        if mod in self.so_modules:
            return AndroidLoader()
        return None
kamathln commented 7 years ago

@endolith I know I am more than 18months late .. But have you checked if the apk generated previously is also getting included in the app? When my app size started doubling every build , I noticed that the previous generated apk was getting bundled into the current apk ..

So, over the last few iterations, my development pattern has morphed somewhat .. I keep all the files that go into the real app in the src directory, and give p4a the src directory as the private folder. The blacklist.txt , project config and other files remain in the project's root directory. I suggest Kivy developers using python-for-android to switch to this pattern. Feels clean.

kamathln commented 7 years ago

@frmdstryr Please look into https://developer.android.com/studio/build/configure-apk-splits.html if you are interested in building multiple architectures. I myself haven't tried it, but is definitely on my todo list.