Loading large custom graph model causes java.lang.OutOfMemoryError on Android (React Native)

VicHofs commented 2 weeks ago

System information

Have I written custom code (as opposed to using a stock example script provided in TensorFlow.js): yes
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Android API 31 (12)
Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: Any Android device
TensorFlow.js installed from (npm or script link): @tensorflowjs/tfjs + @tensorflowjs/tfjs-react-native
TensorFlow.js version (use command below): 4.20.0
Browser version: N/A
Tensorflow.js Converter Version: 3.18.0

Issue

I am trying to import a custom graph model around 50 MB in size with bundleResourceIO (@tensorflow/tfjs-react-native) and loadGraphModel (@tensorflowjs/tfjs) for inference. Building the app with expo run:android successfully builds and installs it on the simulator, but the running app causes it to crash immediately with the following error message:

 ERROR  Your app just crashed. See the error below.
java.lang.OutOfMemoryError: Failed to allocate a 124639368 byte allocation with 25165824 free bytes and 74MB until OOM, target footprint 148421024, growth limit 201326592
(...)

The model loads on the iOS build and causes no other issues. I can load a smaller model (~7MB) with no problem on both platforms, but accuracy suffers significantly.

Minimal example

Here is a repo that replicates this issue. I've omitted my model, so some setup is required, as described in the README.md file.

Here is the logcat output from running the repo above:

08-30 12:05:46.322 10923 10986 W .tfjsgraphissu: Throwing OutOfMemoryError "Failed to allocate a 124639368 byte allocation with 25165824 free bytes and 80MB until OOM, target footprint 141820640, growth limit 201326592" (VmSize 51882480 kB)
08-30 12:05:46.332 10923 10986 E AndroidRuntime: FATAL EXCEPTION: OkHttp Dispatcher
08-30 12:05:46.332 10923 10986 E AndroidRuntime: Process: com.vichofs.tfjsgraphissue, PID: 10923
08-30 12:05:46.332 10923 10986 E AndroidRuntime: java.lang.OutOfMemoryError: Failed to allocate a 124639368 byte allocation with 25165824 free bytes and 80MB until OOM, target footprint 141820640, growth limit 201326592
08-30 12:05:46.332 10923 10986 E AndroidRuntime:        at java.lang.StringFactory.newStringFromBytes(StringFactory.java:86)
08-30 12:05:46.332 10923 10986 E AndroidRuntime:        at java.lang.StringFactory.newStringFromBytes(StringFactory.java:65)
08-30 12:05:46.332 10923 10986 E AndroidRuntime:        at android.util.Base64.encodeToString(Base64.java:458)
08-30 12:05:46.332 10923 10986 E AndroidRuntime:        at com.facebook.react.modules.network.NetworkingModule$2.onResponse(NetworkingModule.java:551)
08-30 12:05:46.332 10923 10986 E AndroidRuntime:        at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519)
08-30 12:05:46.332 10923 10986 E AndroidRuntime:        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
08-30 12:05:46.332 10923 10986 E AndroidRuntime:        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
08-30 12:05:46.332 10923 10986 E AndroidRuntime:        at java.lang.Thread.run(Thread.java:920)

shmishra99 commented 1 week ago

Hi @VicHofs ,

Apologies for the late response. For mobile devices, it is not recommended to use very large models because they can consume a significant amount of memory and potentially cause OutOfMemoryError issues. Memory management on Android and iOS different. You might find this article on Android vs. iOS memory management insightful.

Instead of loading a large model directly on the browser, you can convert your model to TFLite format using quantization. This can significantly reduce the model size and ram usage. You can then use the @tensorflow/tfjs-tflite library to load and run the TFLite model in the browser.

Also you mentioned that you've successfully loaded a smaller model (~7MB) on both devices but it compromises accuracy. Could you please share more details about how you are compressing your model?

Please let me know if there's anything I have missed.

Thank You!!

VicHofs commented 1 week ago

Hi @shmishra99,

Thanks for the response!

Can I load TFLite models for inference on React Native? I'm unsure if I discarded this option earlier because it was not plausible or not, but I know I've considered it in the past.

The smaller model is just another model I trained with fewer parameters and layers for the same purpose, so it is expected to have lower accuracy.

shmishra99 commented 1 week ago

Yes, you can load a tflite model in your react-native application. There are many examples available. You can check out the tflilte react-native examples here.

It's generally better to use a quantized model than one with fewer trained parameters. A quantized model will often perform better, especially on mobile devices.

Thank You!!

github-actions[bot] commented 4 days ago

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

tensorflow / tfjs