search

getodk / collect

ODK Collect is an Android app for filling out forms. It's been used to collect billions of data points in challenging environments around the world. Contribute and make the world a better place! ✨📋✨
https://docs.getodk.org/collect-intro
Other
705 stars 1.36k forks source link

Reduce size of APK #3097

Closed yanokwa closed 5 years ago

yanokwa commented 5 years ago

Historically, the Collect apk has been small (~4MB). Version v1.22.0-beta.1 has grown to 12.3MB. This is likely due to the addition of the Mapbox SDK. 12MB isn't terrible, but for large scale projects it will add up to a lot of time and money.

I'm using ./gradlew clean assembleOdkCollectRelease to build the APK. The attached screenshot is what the Play Store sees as a difference between v1.22.0-beta.0 and v1.22.0-beta.1.

Screen Shot 2019-05-23 at 4 06 57 PM

Some next steps:

yanokwa commented 5 years ago

And for completion, I also get this warning in the Play Store:

Deactivation of this APK will result in your app being available for new installs on fewer types of devices.

It's an odd error because the number of devices on v1.22.0-beta.0 and v1.22.0-beta.1. are the same. I bet it's because of the native platforms change.

zestyping commented 5 years ago

Step 1. Confirmed; Mapbox is the culprit.

Release APK built on master with ./gradlew clean assembleOdkCollectRelease:

-rw-r--r--  1 ping  staff  14240464 23 May 16:53 collect-odkCollectRelease-unsigned-v1.22.0-beta.1.apk

Release APK built with the same command after stubbing out calls to Mapbox classes and removing the two mapbox dependencies from build.gradle:

-rw-r--r--  1 ping  staff  5911033 23 May 16:50 collect-odkCollectRelease-unsigned-v1.22.0-beta.1-dirty.apk
zestyping commented 5 years ago

The size change is 5.9 MB to 14.2 MB — an increase of 8.3 MB or a factor of 2.4x.

yanokwa commented 5 years ago

https://proandroiddev.com/reducing-apk-size-by-using-abi-filters-and-apk-split-74a68a885f4e and the refs at the bottom have good info on what we should include/exclude if we want to build one APK.

zestyping commented 5 years ago

Here's the breakdown of the size increase.

2019-05-24 17-35-10

Pretty much all of the increase in download size (+7.9 MB) consists of the new native binaries (+7.3 MB) and a small increase in classes/resources (+0.4 MB), which I assume to be the Java side of the Mapbox SDK.

zestyping commented 5 years ago

Based on https://developer.android.com/ndk/guides/abis.html, it seems to me that:

  1. We could include only the armeabi binary alone, and it would still work on all phones we currently support, i.e. API level 16+ (though it wouldn't take advantage of the extra capabilities of ARMv7 and ARMv8 devices).
  2. We could include only the armeabi-7a binary alone, and it would work on all phones with API level 17+ (though it wouldn't take advantage of the extra capabilities of ARMv8 devices).
  3. We don't need arm64-v8a or x86_64 because all devices that run arm64-v8a can also run armeabi-7a, and all devices that run x86_64 can also run x86.
  4. We (probably?) don't need to include the x86 binary in the release because it's only used by emulators and by a small (and decreasing) number of devices. But maybe we could check our analytics to see if there are any x86 devices in there?

@yanokwa This differs from your assessment—could you confirm if I am reading that right?

zestyping commented 5 years ago

If I drop osmdroid from the build:

-rw-r--r--  1 ping  staff  14083793 24 May 11:23 collect-odkCollectRelease-unsigned-v1.22.0-beta.1-dirty.apk

The size only reduces from 14240464 to 14083793, a savings of 156 kB or 1.1%. 😕 Native libraries are huge!

zestyping commented 5 years ago

Here is a list of all the phones from phonedb.net that have an Intel or AMD processor, run Android 4.1 or higher, and have GPS capability:

x86-models.txt

There are 155 models. Consolidating similar models, this list simplifies down to:

Acer Iconia
Acer Predator
Allview Viva
Asus FonePad
Asus MeMO
Asus PadFone
Asus Transformer
Asus ZenFone
Asus ZenPad
Axioo Sofia
Cherry Mobile MAIA
Dell Venue
Geeksphone Revolution
Huawei Porsche
Intel Education
Lenovo IdeaPhone
Lenovo K80
Lenovo P90
Lenovo TAB S8-50 LTE
Lenovo YT3-X90
Lenovo Yoga Book X91
Lenovo Yoga Tablet 2 1050
Micromax P666
Panasonic Toughpad FZ
Samsung GT-P5200
Samsung GT-P5210
Samsung GT-P5220
Starmobile Engage
Teclast P89s
Vido Yuandao M6
Xiaomi Mi Pad 2
ZTE Grand X 2

Using this list, I created a Report named "x86 devices, 2018-2019" in Google Analytics with the following conditions:

The report shows 25 models used since January 1, 2018, accounting for 1,026 users (0.10% of users) and 254,910 sessions (0.10% of sessions):

2019-05-24 20-36-24

yanokwa commented 5 years ago

@zestyping I'm still thinking about your earlier question at https://github.com/opendatakit/collect/issues/3097#issuecomment-495741430. While I do that, do you have a sense for how bad the performance difference is between say arm64-v8a and armeabi-7a? Is is the type of thing that the average user would notice?

yanokwa commented 5 years ago

Well, ignore my previous comment. According to https://developer.android.com/distribute/best-practices/develop/64-bit, starting August 1, 2019, we will need to ship with a 64-bit library if we use native code.

So, we have to include arm64-v8a and one of armeabi-v7a or armeabi depending on what API level we want to support. If we include x86, then we have to include x86_64.

zestyping commented 5 years ago

Aaaaahh, FU, Google. What a useless waste of space and bandwidth for people who don't need the (probably minimal) speedup that going from 32-bit to 64-bit delivers. Yet again, thinking only of those privileged with ubiquitous and plentiful Internet bandwidth, who don't pay by the megabyte, and who can afford to throw away their phones every couple of years.

yanokwa commented 5 years ago

The Google giveth and the Google taketh away!

One option is that we publish APKs for every ABI. That's a tough thing to do logistically. Uploading multiple APKs, keeping the versionCodes different, sound like a nightmare. Maybe there are tools that automate the process?

Another option is that we use App Bundles. This option feels more promising but I have done zero research so who knows what dragons lay ahead.

zestyping commented 5 years ago

The problem with both those solutions is that they totally break Bluetooth app transfer.

zestyping commented 5 years ago

(Or local serving of an APK file, which is super useful in the field.)

zestyping commented 5 years ago

Okay, I've finished making all these builds so we can compare them:

-rw-r--r--  1 ping  staff   5911033 24 May 11:51 collect-v1.22.0b.no-mapbox.apk
-rw-r--r--  1 ping  staff   6476634 24 May 13:44 collect-v1.22.0b.no-mapbox-native-code.apk
-rw-r--r--  1 ping  staff   8222698 24 May 14:24 collect-v1.22.0b.v7a.apk
-rw-r--r--  1 ping  staff   8413860 24 May 15:34 collect-v1.22.0b.v8a.apk
-rw-r--r--  1 ping  staff   8515442 24 May 15:40 collect-v1.22.0b.x86.apk
-rw-r--r--  1 ping  staff   8518530 24 May 15:36 collect-v1.22.0b.x86_64.apk
-rw-r--r--  1 ping  staff  10159962 24 May 14:17 collect-v1.22.0b.v7a-v8a.apk
-rw-r--r--  1 ping  staff  10261538 24 May 14:21 collect-v1.22.0b.v7a-x86.apk
-rw-r--r--  1 ping  staff  14083793 24 May 11:52 collect-v1.22.0b.v7a-v8a-x86-x86_64.no-osmdroid.apk
-rw-r--r--  1 ping  staff  14240464 24 May 11:50 collect-v1.22.0b.v7a-v8a-x86-x86_64.apk

Our baseline is the no-mapbox build, at 5.9 MB. The no-mapbox-native-code build contains the Mapbox Java classes but not the native libraries, so we can see the impact of each native library.

I was going to recommend the v7a build (8.2 MB), which is a tolerable +2.3 MB (1.4x) increase, runnable for 99.90% of our users.

But Google wants to use its monopoly power to force us to distribute the v7a-v8a build (10.2 MB) or v7a-v8a-x86-x86_64 (14.2 MB), which are 1.7x bigger and 2.4x bigger respectively.

zestyping commented 5 years ago

Another option we could consider is that we could publish two APKs: v7a and v7a-v8a. We publish the v7a-v8a version only to satisfy Google's pointless requirement. Both versions are safe to distribute over Bluetooth, but the v7a-v8a version is only a formality. We distribute v7a as the true, official version.

(It might even be possible to write the v7a-v8a version so that it downloads and replaces itself with the v7a version upon installation, ensuring that Bluetooth transfers are fast.)

I imagine the ability to run 32-bit apps is unlikely to go away for 4–5 years. The Play Store will continue to serve 32-bit apps to all devices until August 2021; after that, 64-bit devices won't see them anymore, but 32-bit apps will still be served to 32-bit devices indefinitely. Even when 2021 comes, there is no reason to actually make 64-bit phones that suddenly can't run apps > 2 years old.

zestyping commented 5 years ago

Okay, here are all the options I can think of. We don't have to decide on the long-term plan immediately; we can choose to postpone some or all of the decisions until v1.23 or until August.

Options we have now:

APKs for v1.22 release APK sizes Who gets Mapbox? APK redistributable?
1. no Mapbox (postpone) 5.9 MB (1.0x) no one yes
1. v7a only 8.2 MB (1.4x) 99.90% of users yes
1. v7a+x86 10.3 MB (1.7x) everyone yes
1. v7a+v8a 10.2 MB (1.7x) 99.90% of users yes
1. v7a+v8a+x86+x86_64 14.2 MB (2.4x) everyone yes
1. v7a
2. v8a		 8.2 MB (1.4x)
8.4 MB (1.4x)		 99.90% of users no
1. v7a
2. v7a+v8a		 8.2 MB (1.4x)
10.2 MB (1.7x)		 99.90% of users yes
1. v7a
2. v8a
3. x86
4. x86_64		 8.2 MB (1.4x)
8.4 MB (1.4x)
8.4 MB (1.4x)
8.5 MB (1.4x)		 everyone no

Options we will have after August 2019:

Release APKs APK sizes Who gets Mapbox? APK redistributable?
1. v7a+v8a 10.2 MB (1.7x) 99.90% of users yes
1. v7a+v8a+x86+x86_64 14.2 MB (2.4x) everyone yes
1. v7a
2. v8a		 8.2 MB (1.4x)
8.4 MB (1.4x)		 99.90% of users no
1. v7a
2. v7a+v8a		 8.2 MB (1.4x)
10.2 MB (1.7x)		 99.90% of users yes
1. v7a
2. v8a
3. x86
4. x86_64		 8.2 MB (1.4x)
8.4 MB (1.4x)
8.4 MB (1.4x)
8.5 MB (1.4x)		 everyone no
zestyping commented 5 years ago

@yanokwa Thoughts on next steps? I think this should be your call. (My personal leaning is toward releasing a v7a-only APK for now so we start getting feedback on Mapbox right away, but pretty happy to go with anything you decide.) Let me know if there's any more data I can get for you that would factor into your decision.

lognaturel commented 5 years ago

Quick note that the size in the Play Store is a little bit smaller than what gets uploaded (because ???). Right now the beta is showing up as 11.53mb.

What happens to users on API 16 with v7a only? Are they cut off from updating Collect just like the x86 users or can they update but Mapbox won't work?

yanokwa commented 5 years ago

I think we should ship v7a+x86 to get everyone for now, communicate in the release notes why the app size has gone up, and communicate about upcoming file size changes and potential dropping of x86. Reaching everyone is important because there are good features in this release independent of maps: logic in field lists and track changes.

There might be a performance hit on 64-bit devices with this approach, but I doubt it'll be severe enough for people to notice (I have an email out to Mapbox asking about this). Also, since Mapbox isn't the default, it shouldn't affect most people.

I don't love that we are burdening v7a folks with an extra 2 MB just to support x86, but they'll have to download even more anyway in August with the 64-bit requirement. If bandwidth is a huge problem, we still let people download from GitHub and sideload.

Also, at the end of the day, even 10-15 MB isn't terrible. Magpi is 23 MB, Commcare is 20 MB, DHIS2 Capture is 15 MB.

Question that is blocking the release is the same as @lognaturel's. What happens to API 16 people?

zestyping commented 5 years ago

@lognaturel Good point about API 16.

You're right, this is the big open question, and it bears on both the API 16 question and on the decision to include x86. I had assumed that the APK without the right native libraries would still run but only fail when trying to make native calls into the native library (thus API 16 and x86 users would only miss out on Mapbox but everything else would still work) because that's the obviously more sane behaviour, but that probably means we should be cautious that it might not be the case.

zestyping commented 5 years ago

To clarify—it seems almost certain that it's possible to deploy an APK that gets served to all users with only the Mapbox part not working for API 16 and x86 users. It's just a question of how hard the Play Store is going to make it to accomplish that. (In the worst case, we could publish a fat APK containing armeabi and arm64-v8a and x86 libraries that are full of empty stubs.)

zestyping commented 5 years ago

Okay, I've confirmed that I can build and install an APK that lacks the native library for a given platform, and we can detect this in the code (so we could gracefully disable the Mapbox option in such a case). But I haven't confirmed that the Play Store will serve it.

@yanokwa I'll confirm this later today if I can. But you don't need to block on that confirmation—if you want to proceed, the quickest way around the problem is to ship with armeabi and x86, and then we'll support everybody including API 16 and x86 users (armeabi supports everyone, armeabi-v7a supports only API 17+).

yanokwa commented 5 years ago

If we can ship armeabi and x86, why not just do that until August? Is the performance of armeabi-v7a that much worse?

zestyping commented 5 years ago

If we are okay with the size increasing to ~10 MB, I see no reason not to do that until August.

The only performance impact this could possibly have is that the Mapbox library might run a bit slower than a hypothetical build with armeabi-v7a. But that's in comparison to a build that doesn't exist, and there's no impact on the rest of the app because everything else is in Java.

zestyping commented 5 years ago

Oops. I spoke too soon. Mapbox dropped support for armeabi in May 2018.

https://github.com/mapbox/mapbox-gl-native/pull/11458

zestyping commented 5 years ago

Okay. I tried publishing an app on Google Play that contains only an x86 binary and went through the whole process. It finally appeared in the Play Store. On my phone, the Play Store says "This app isn't compatible with your device anymore" and there's no Install button—even though the app runs just fine on my ARM-based phone. 😒

I expect that API 16 users will have this experience if we ship with armeabi-v7a and x86. We should confirm by trying this out, though. Do we have any API 16 devices we can test with? I can publish a test app with armeabi-v7a and x86 and we can try it out.

If the Play Store refuses to serve to an API 16 device, then we will have to either add stubs for armeabi, cut API 16 users off from updates, or drop Mapbox support. I hate you Google.

zestyping commented 5 years ago

Oh, I have one more idea. What if we drop in a zero-byte file for armeabi/libmapbox-gl.so? Hmm...

lognaturel commented 5 years ago

I also expect that API 16 users would not be able to upgrade if the build only has armeabi-v7a and x86 libs. I have a device I can confirm with.

What if we drop in a zero-byte file for armeabi/libmapbox-gl.so

This is a really good idea. Any results?

On another note, do we know for sure that shipping App Bundles precludes bluetooth transfer? Is the fastest way to confirm one way or another to try it?

yanokwa commented 5 years ago

My new ideal would be ship armeabi-v7a, put in zero-byte file or stubs for the rest, and disable Mapbox option (or throw up a warning) for API 16, x86.

zestyping commented 5 years ago

@yanokwa Okay, I did an exploration of this today that led me down several garden paths.

I'll try to write a shorter summary soon, but here's a description of everything for now—apologies for the length! See the last bullet point for the TL;DR on what we can do now.

lognaturel commented 5 years ago

Thanks for that deep dive, @zestyping. I know it's probably a lot more than you wanted to know about all this. No summary needed for me.

We would be cutting armeabi (users of devices with ARMv5 and ARMv6 CPUs) from Play Store updates

To be explicit about one thing that you explained to me that I don't think has made it into this thread -- while it's likely that there are some devices that run Collect from Android API 16 and ARM processors that don't support the armv7 instruction set, it's probably not very many. For example, The Alcatel M'Pop I have that runs Android 4.1.1 has a processor that supports the armv7 instruction set. So not having an armeabi native lib does not affect everyone on API 16, only those with armv5 or armv6 procs.

build a binary that ships armeabi-v7a + x86 — for v1.22

Alternately we can ship separate armeabi-v7a and x86 binaries this one time.

zestyping commented 5 years ago

That's correct. According to https://source.android.com/compatibility/4.4/android-4.4-cdd.html#section-3.3.1, all Android devices from 4.4 up (API level 19+) are required to support armeabi-v7a. So the set of devices that we currently push updates to, that would be affected by shipping an APK to Google Play that includes some native libraries but not an armeabi library, includes only devices running API levels 16, 17, or 18 (Android 4.1 through 4.3, all the versions named "Jelly Bean"), and would only be the non-ARMv7 subset of those devices.

Some of those devices are listed at https://forum.xda-developers.com/showthread.php?t=1596800 . The ARMv5 and ARMv6 lists look long, but I expect they would be a small minority of devices still in use today, even among our target audience.

zestyping commented 5 years ago

If anyone is curious to check a particular phone, the fastest way to find out the supported architectures is

adb shell getprop | grep cpu
yanokwa commented 5 years ago

I'm leaning towards using the zstd library trick to build a single binary for armeabi-v7a and x86. It feels low risk to me. @zestyping, any reason why you didn't recommend it as the approach for v1.22?

zestyping commented 5 years ago

Adding zero-byte stub binaries was indeed the main goal of the entire exercise. The only reason I didn't do that last step is that I had spent the whole day exploring all these possibilities already, it sounded like we wanted to release to go out today or tomorrow, and I didn't have the energy to figure out how to build our own package with stub binaries in the time left—I didn't realize it was so trivial to create an .aar file. Looks like I was closer to the finish line than I realized—glad you figured out the last step!

yanokwa commented 5 years ago

@zestyping You did all the hard work! Glad I could chip in a little bit.

zestyping commented 5 years ago

Gradle sucked all the life out of me! I'm glad you tagged in.