Apk size is too large after using this library

[jochendev]

I like objectbox very much in android native, but after introducing this dart library in flutter, the apk size is very large, which has increased by about 15MB (total size is 33MB. after excluding objectbox-dart is 18MB). Is there any way to reduce the size? I am using the realease build. If use debug build, the size is double.

[greenrobot]

2 MB would be expected for an APK covering all CPUs (see FAQ)... :thinking:

Can you please run APK Analyzer on your 33MB APK and report back where those additional 15MB come from?

[jochendev]

2 MB would be expected for an APK covering all CPUs (see FAQ)... 🤔

Can you please run APK Analyzer on your 33MB APK and report back where those additional 15MB come from?

without library & release added library & release added library & debug 😂😂😂

[greenrobot]

OK, so to summarize:

• Downloadable size went up by only 1.5 MB
• libflutter.so and libapp.so went from 12.9 MB up by 19 MB to 31,9 MB

Not sure what caused this 19MB increase; it seems far too high. Maybe we can summon e.g. @dcharkes to ask if this might be related to Dart FFI?

@qiushouchen Can you supply the versions of Dart and Flutter that you used?

PS.: total APK size up by 23 MB with only 1.5 MB increased for download is very weird. If 1.5MB is about the compressed size of ObjectBox, this hints at that the additional data for libapp&libflutter is "phantom data", that is "compressed away". :thinking:

[jochendev]

OK, so to summarize:

• Downloadable size went up by only 1.5 MB
• libflutter.so and libapp.so went from 12.9 MB up by 19 MB to 31,9 MB

Not sure what caused this 19MB increase; it seems far too high. Maybe we can summon e.g. @dcharkes to ask if this might be related to Dart FFI?

@qiushouchen Can you supply the versions of Dart and Flutter that you used?

PS.: total APK size up by 23 MB with only 1.5 MB increased for download is very weird. If 1.5MB is about the compressed size of ObjectBox, this hints at that the additional data for libapp&libflutter is "phantom data", that is "compressed away". 🤔

Yes dart 2.7.0 flutter 1.12.13+hotfix.8 channel stable

[dcharkes]

libflutter.so and libapp.so went from 12.9 MB up by 19 MB to 31,9 MB

That is curious indeed. libapp.so could be from FFI things. Can you give me a small reproduction project (or steps) for this? @mkustermann how can we analyze what is in a libapp.so?

libflutter.so is unlikely due to FFI, I'm not aware of FFI influencing libflutter.so at all. My best guess is that the objectbox shared library is statically linked into libflutter.so.

I'm a bit confused by libobjectbox-jni.so. When using Dart FFI (bridge between Dart and C), there's no use of JNI (bridge between Java and C). So why is there some JNI related things included?

[jochendev]

libflutter.so and libapp.so went from 12.9 MB up by 19 MB to 31,9 MB

That is curious indeed. libapp.so could be from FFI things. Can you give me a small reproduction project (or steps) for this? @mkustermann how can we analyze what is in a libapp.so?

libflutter.so is unlikely due to FFI, I'm not aware of FFI influencing libflutter.so at all. My best guess is that the objectbox shared library is statically linked into libflutter.so.

I'm a bit confused by libobjectbox-jni.so. When using Dart FFI (bridge between Dart and C), there's no use of JNI (bridge between Java and C). So why is there some JNI related things included?

I tried to create a new empty flutter project and introduced the objectbox-dart library. The final packaged apk is also very large. I do n’t know if there is something wrong with my environment, but I did n’t find anything abnormal when I run ‘flutter doctor’.

[vaind]

I'm a bit confused by libobjectbox-jni.so. When using Dart FFI (bridge between Dart and C), there's no use of JNI (bridge between Java and C). So why is there some JNI related things included?

The .so provides JNI functions in addition to plain C (objectbox-c) methods, only the latter being used by objectbox-dart. We're reusing the android build from objectbox-java, because objectbox-c doesn't have an android release yet and the overhead of the JNI part of the libobjectbox-jni.so isn't so large anyway.

[vaind]

Can you give me a small reproduction project (or steps) for this?

Created a minimal sample at https://github.com/vaind/objectbox-flutter-size using:

1. flutter create with_objectbox and flutter create without_objectbox
2. adding objectbox to with_objectbox/podspec.yaml

   
   dependencies:
   objectbox: ^0.6.2

dev_dependencies: build_runner: ^1.0.0 objectbox_generator: ^0.6.2

3. ran `flutter pub get` in both 
4. and built (release) using `flutter build apk` and `flutter build appbundle`

| project | apk | appbundle |
|----------|-------|----------------|
| with OBX |  40.7 MB | 17.3 MB |
| without OBX | 15.2 MB | 15.3 MB |

So it seems only APK is much larger than expected, appbundle is in line with the expected increase of about 2 MB for objectbox, FFI, etc.

Flutter version:

[✓] Flutter (Channel stable, v1.12.13+hotfix.8, on Linux, locale en_US.UTF-8) [✓] Android toolchain - develop for Android devices (Android SDK version 29.0.2)

[greenrobot]

@dcharkes, just to clarify this:

libflutter.so is unlikely due to FFI, I'm not aware of FFI influencing libflutter.so at all. My best guess is that the objectbox shared library is statically linked into libflutter.so.

Because we see libobjectbox-jni.so separately, it should not be statically linked, right? As vaind already wrote, libobjectbox-jni.so contains both JNI and the plain C API. Our Flutter binding only uses the plain C API.

Or, does the presence of a JNI API alone trigger something in Dart/Flutter?

Please let us know if we should move this over to Dart issues. Thanks.

[jochendev]

@dcharkes, just to clarify this:

libflutter.so is unlikely due to FFI, I'm not aware of FFI influencing libflutter.so at all. My best guess is that the objectbox shared library is statically linked into libflutter.so.

Because we see libobjectbox-jni.so separately, it should not be statically linked, right? As vaind already wrote, libobjectbox-jni.so contains both JNI and the plain C API. Our Flutter binding only uses the plain C API.

Or, does the presence of a JNI API alone trigger something in Dart/Flutter?

Please let us know if we should move this over to Dart issues. Thanks.

Some new discoveries.:

When I add the following code to the application tag.

Apk size is normal.

android:extractNativeLibs="true" tools:replace="android:extractNativeLibs"

[greenrobot]

Interesting find. Because this enables compression of the .so files, it may be in line with what I've written earlier:

If 1.5MB is about the compressed size of ObjectBox, this hints at that the additional data for libapp&libflutter is "phantom data", that is "compressed away".

The two Flutter related libs seem blown up with "nonsense", which is compressed away...

[dcharkes]

@qiushouchen that's great news! In which file in a Flutter project is that? And is that file pre-generated by Flutter (e.g. do we need to change Flutter)? Or is it something developers can easily change (e.g. we should add it to Flutter+FFi documentation)?

[jochendev]

@qiushouchen that's great news! In which file in a Flutter project is that? And is that file pre-generated by Flutter (e.g. do we need to change Flutter)? Or is it something developers can easily change (e.g. we should add it to Flutter+FFi documentation)?

The file I modified is AndroidManifest.xml under android project.

[greenrobot]

@dcharkes Lets's be careful calling it great news. Flutter/Dart generated .so files seem filled up with nonsense, which happens to be very compressible. The flag hides the problem as it triggers compression of .so in APK files. HOWEVER, those files will be uncompressed on the device. Same problem, just less visible.

[mraleph]

@greenrobot you can notice that libobjectbox-jni.so is filled with the same "compressible nonsense". It goes from 1.5 megabytes to 500 kilobytes once it is compressed. That's compression ratio of 3.

For comparison:

• libflutter.so (it is just a C++ binary) has compression ratio of 2.25
• libapp.so has compression rate of 2.8

I don't think expectation that native code should be non-compressable is realistic.

[mkustermann]

Regarding APK size: The raw APK size does not say that much about download size: Devices download the apks brotli compressed. The devices then store the apk on device and decompress shared libraries if and only if they are compressed. So overall it's actually better to have the shared libraries uncompressed in the APK.

For updates android uses binary diffing to only download compressed binary diff. There's a size estimator for diffs: https://github.com/googlesamples/apk-patch-size-estimator

Created a minimal sample at https://github.com/vaind/objectbox-flutter-size using:

On my machine:

% ls -h $(find . -name '*.so' | grep -v '[.]dart_tool' | grep 'app.so')
3.5M ./with_objectbox/build/aot/app.so
3.5M ./with_objectbox/build/app/intermediates/flutter/release/arm64-v8a/app.so
3.6M ./with_objectbox/build/app/intermediates/flutter/release/armeabi-v7a/app.so
3.5M ./with_objectbox/build/app/intermediates/flutter/release/x86_64/app.so
3.5M ./with_objectbox/build/app/intermediates/merged_native_libs/release/out/lib/arm64-v8a/libapp.so
3.6M ./with_objectbox/build/app/intermediates/merged_native_libs/release/out/lib/armeabi-v7a/libapp.so
3.5M ./with_objectbox/build/app/intermediates/merged_native_libs/release/out/lib/x86_64/libapp.so
3.5M ./with_objectbox/build/app/intermediates/stripped_native_libs/release/out/lib/arm64-v8a/libapp.so*
3.6M ./with_objectbox/build/app/intermediates/stripped_native_libs/release/out/lib/armeabi-v7a/libapp.so*
3.5M ./with_objectbox/build/app/intermediates/stripped_native_libs/release/out/lib/x86_64/libapp.so*
3.5M ./without_objectbox/build/aot/app.so
3.5M ./without_objectbox/build/app/intermediates/flutter/release/arm64-v8a/app.so
3.6M ./without_objectbox/build/app/intermediates/flutter/release/armeabi-v7a/app.so
3.5M ./without_objectbox/build/app/intermediates/flutter/release/x86_64/app.so
3.5M ./without_objectbox/build/app/intermediates/merged_native_libs/release/out/lib/arm64-v8a/libapp.so
3.6M ./without_objectbox/build/app/intermediates/merged_native_libs/release/out/lib/armeabi-v7a/libapp.so
3.5M ./without_objectbox/build/app/intermediates/merged_native_libs/release/out/lib/x86_64/libapp.so
3.5M ./without_objectbox/build/app/intermediates/stripped_native_libs/release/out/lib/arm64-v8a/libapp.so*
3.6M ./without_objectbox/build/app/intermediates/stripped_native_libs/release/out/lib/armeabi-v7a/libapp.so*
3.5M ./without_objectbox/build/app/intermediates/stripped_native_libs/release/out/lib/x86_64/libapp.so*

So the size of the compiled dart application is almost the same. (similar for libflutter.so)

The diff in apks:

% unzip -l with_objectbox/build/app/outputs/apk/release/app-release.apk
Archive:  with_objectbox/build/app/outputs/apk/release/app-release.apk
  Length      Date    Time    Name
---------  ---------- -----   ----
...
---------                     -------
 42762435                     366 files
% unzip -l without_objectbox/build/app/outputs/apk/release/app-release.apk
Archive:  without_objectbox/build/app/outputs/apk/release/app-release.apk
  Length      Date    Time    Name
---------  ---------- -----   ----
...
---------                     -------
 37174573                     358 files

The difference in bytes is around 5.3 MB and is mainly explained by libobjectbox-jni.so being added in various architectures.

[greenrobot]

@mraleph I'm not sure if you caught the initial statements. In summary, after adding ObjectBox "libflutter.so and libapp.so went from 12.9 MB up by 19 MB to 31,9 MB" and "only 1.5 MB increased for download is very weird. If 1.5MB is about the compressed size of ObjectBox, this hints at that the additional data for libapp&libflutter is "phantom data", that is "compressed away"". Edit: Totally agree; the unexpected thing was that the flutter libs were compressed before using FFI and uncompressed when FFI was activated.

@mkustermann You have seen the original reported sizes? What could cause the difference?

[greenrobot]

OK, that was unexpected:

Using dart:ffi inverts this default setting, and stops the compression of libflutter.so, libapp.so, and any user-added shared objects.

[mkustermann]

What could cause the difference?

The major APK size difference must be compressed vs non-compressed apk (or the so files therein)

[greenrobot]

@mkustermann Yes, https://github.com/dcharkes/website/commit/2f4b73ddb1731c0af94da975c44406f9ef139bf9 explains it all. By adding an .so, the compression setting is inverted. I did not see this coming. It's a surprising behavior: as you can see people see this as a fault of the library, which happens to have native libs.

[mraleph]

@greenrobot

@mraleph I'm not sure if you caught the initial statements.

I did. But I think you now understand what I meant - I was commenting under assumption that you have seen https://github.com/flutter/flutter/issues/53738, which explains what happens. There is no phantom data, it's just compression settings. Sorry for not being clear.

It's a surprising behavior: as you can see people see this as a fault of the library, which happens to have native libs.

Agreed, though in this particular case it does seem to be due to ObjectBox choices. See https://docs.objectbox.io/faq#how-much-does-objectbox-add-to-my-apk-size. Quoting:

The raw file (APK or AAB) size increases around 5.3 MB. This is because ObjectBox adds extractNativeLibs="false" to your AndroidManifest.xml as recommended by Google. This turns off compression. However, this allows Google Play to optimally compress the APK before downloading it to each device (see download size above) and reduces the size of your app updates (on Android 6.0 or newer). Read this Android developers post for details. It also avoids issues that might occur when extracting the libraries.

[greenrobot]

Now that's a good twist... Thanks for pointing out what we are doing. :rofl:

Thanks for getting to the bottom of it! Very much appreciated! :+1:

Now, we'll do some thinking what we shall do here... Go with Google recommendations or with Flutter's... :grin:

[mraleph]

Go with Google recommendations or with Flutter's

FWIW we are talking with Flutter team about switching to Flutter APKs to use uncompressed libs as per Android recommendations.

[greenrobot]

For now, we handle this via a new FAQ entry: https://github.com/objectbox/objectbox-dart/commit/8eddac98b873dc61a93fe06bd98a09454659ec25

@mraleph yeah, I'd welcome that. Along with another size optimization round maybe? :grin: Thanks, over and out.

[daixianceng]

@dcharkes, just to clarify this:

libflutter.so is unlikely due to FFI, I'm not aware of FFI influencing libflutter.so at all. My best guess is that the objectbox shared library is statically linked into libflutter.so.

Because we see libobjectbox-jni.so separately, it should not be statically linked, right? As vaind already wrote, libobjectbox-jni.so contains both JNI and the plain C API. Our Flutter binding only uses the plain C API. Or, does the presence of a JNI API alone trigger something in Dart/Flutter? Please let us know if we should move this over to Dart issues. Thanks.

Some new discoveries.:

When I add the following code to the application tag.

Apk size is normal.

android:extractNativeLibs="true" tools:replace="android:extractNativeLibs"

After I upgrade gradle from 3.6.0 to 4.1.0, the apk size went from 22 MB up to 47.5 MB. This answer solved my problem.