MLKitBarcodeScanner OutOfMemoryError after +- 50 Scans

[EinatGit]

Hello,

We are using MLKitBarcodeScanner for scanning QR codes with NativeScript Angular v6.5.1. We managed to scan +- 50 QR in a row with 2-3 secondes of pause between them, and then the app freeze and we get this error:

System.err: java.lang.OutOfMemoryError: Failed to allocate a 1048592 byte allocation with 953272 free bytes and 930KB until OOM, max allowed footprint 536870912, growth limit 536870912 System.err: at java.util.Arrays.copyOf(Arrays.java:3164) System.err: at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118) System.err: at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) System.err: at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153) System.err: at android.graphics.YuvImage.nativeCompressToJpeg(Native Method) System.err: at android.graphics.YuvImage.compressToJpeg(YuvImage.java:141) System.err: at com.google.android.gms.internal.firebase_ml.zzqo.zza(Unknown Source:10) System.err: at com.google.firebase.ml.vision.common.FirebaseVisionImage.zzai(Unknown Source:101) System.err: at com.google.firebase.ml.vision.common.FirebaseVisionImage.zznl(Unknown Source:51) System.err: at com.google.firebase.ml.vision.common.FirebaseVisionImage.getBitmap(Unknown Source:46) System.err: at com.tns.Runtime.callJSMethodNative(Native Method) System.err: at com.tns.Runtime.dispatchCallJSMethodNative(Runtime.java:1286) System.err: at com.tns.Runtime.callJSMethodImpl(Runtime.java:1173) System.err: at com.tns.Runtime.callJSMethod(Runtime.java:1160) System.err: at com.tns.Runtime.callJSMethod(Runtime.java:1138) System.err: at com.tns.Runtime.callJSMethod(Runtime.java:1134) System.err: at com.tns.gen.com.google.android.gms.tasks.OnSuccessListener.onSuccess(OnSuccessListener.java:19) System.err: at com.google.android.gms.tasks.zzn.run(Unknown Source:4) System.err: at android.os.Handler.handleCallback(Handler.java:873) System.err: at android.os.Handler.dispatchMessage(Handler.java:99) System.err: at android.os.Looper.loop(Looper.java:193) System.err: at android.app.ActivityThread.main(ActivityThread.java:6898) System.err: at java.lang.reflect.Method.invoke(Native Method) System.err: at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:537) System.err: at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:858) JS: topmost() is deprecated. Use Frame.topmost() instead. JS: sizePair.pictureSize: 4608x3456 JS: sizePair.previewSize: 1440x1080 JS: Error in Firebase MLKit's runCamera function: Error: java.lang.OutOfMemoryError: Failed to allocate a 2332816 byte allocation with 1427120 free bytes and 1393KB until OOM, max allowed footprint 536870912, growth limit 536870912 JS: topmost() is deprecated. Use Frame.topmost() instead. JS: sizePair.pictureSize: 4608x3456 JS: sizePair.previewSize: 1440x1080 JS: Error in Firebase MLKit's runCamera function: Error: java.lang.OutOfMemoryError: Failed to allocate a 2332816 byte allocation with 745968 free bytes and 728KB until OOM, max allowed footprint 536870912, growth limit 536870912 JS: sizePair.pictureSize: 4608x3456 JS: sizePair.previewSize: 1440x1080 JS: Error in Firebase MLKit's runCamera function: Error: java.lang.OutOfMemoryError: Failed to allocate a 2332816 byte allocation with 691768 free bytes and 675KB until OOM, max allowed footprint 536870912, growth limit 536870912

---

We've used the pause attribute to prevent scanning without a break (without using pause - the app crashes after approximately 8 scans in a row). We got this error when using different kinds of android devices: android 10 with 4GB RAM, android 9 with 8GB RAM, Android 6 with 4GB RAM. We also have this issue on iOS Iphone 7 (2 GB RAM).

We've added

android:largeHeap="true"

in the manifest, and :

dexOptions {
    javaMaxHeapSize "4g"
  }

in app.gradle

More Information: package.json:

{
  "nativescript": {
    "id": "org.nativescript.MYID",
    "tns-ios": {
      "version": "6.2.0"
    },
    "tns-android": {
      "version": "6.5.3"
    }
  },
  "description": "NativeScript Application",
  "license": "SEE LICENSE IN <your-license-filename>",
  "repository": "<fill-your-repository-here>",
  "dependencies": {
    "@angular/animations": "~8.2.0",
    "@angular/common": "~8.2.0",
    "@angular/compiler": "~8.2.0",
    "@angular/core": "~8.2.0",
    "@angular/forms": "~8.2.0",
    "@angular/http": "8.0.0-beta.10",
    "@angular/platform-browser": "~8.2.0",
    "@angular/platform-browser-dynamic": "~8.2.0",
    "@angular/router": "~8.2.0",
    "@ngx-translate/core": "^11.0.1",
    "@ngx-translate/http-loader": "^4.0.0",
    "@nstudio/nativescript-loading-indicator": "^1.0.0",
    "nativescript-angular": "^8.20.4",
    "nativescript-background-http": "^3.4.0",
    "nativescript-camera": "^4.4.0",
    "nativescript-datetimepicker": "^1.2.2",
    "nativescript-imagecropper": "^3.0.0",
    "nativescript-imagepicker": "^7.1.0",
    "nativescript-nfc": "4.0.1",
    "nativescript-permissions": "^1.3.8",
    "nativescript-plugin-firebase": "9.0.2",
    "nativescript-theme-core": "~1.0.4",
    "nativescript-ui-chart": "^7.1.1",
    "nativescript-ui-sidedrawer": "^8.0.1",
    "reflect-metadata": "~0.1.10",
    "rxjs": "^6.4.0",
    "tns-core-modules": "^6.3.2",
    "zone.js": "^0.9.1"
  },
  "devDependencies": {
    "@angular/compiler-cli": "~8.2.0",
    "@nativescript/schematics": "~0.5.0",
    "@ngtools/webpack": "~8.2.0",
    "@types/jasmine": "^3.5.11",
    "nativescript-dev-webpack": "^1.4.1",
    "tns-platform-declarations": "6.0.1",
    "typescript": "~3.5.3"
  },
  "readme": "NativeScript Application"
}

AndroidManifest:

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
    package="__PACKAGE__"
    android:versionCode="20003"
    android:versionName="2.03">

    <supports-screens
        android:smallScreens="true"
        android:normalScreens="true"
        android:largeScreens="true"
        android:xlargeScreens="true"/>

    <uses-sdk
        android:minSdkVersion="17"
        android:targetSdkVersion="__APILEVEL__"/>

    <uses-permission android:name="android.permission.CAMERA"/>
    <uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE"/>
    <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>
    <uses-permission android:name="android.permission.INTERNET"/>

    <uses-feature android:name="android.hardware.camera" android:required="false" />
    <uses-feature android:name="android.hardware.camera.autofocus" android:required="false" />
    <meta-data
    android:name="com.google.firebase.ml.vision.DEPENDENCIES"
    android:value="barcode"/>

    <application
        android:name="com.tns.NativeScriptApplication"
        android:allowBackup="true"
        android:icon="@drawable/icon"
        android:label="@string/app_name"
        android:requestLegacyExternalStorage="true"
        android:largeHeap="true"
        android:theme="@style/AppTheme">

        <activity
            android:name="com.tns.NativeScriptActivity" 
            android:label="@string/title_activity_kimera"
            android:configChanges="keyboardHidden|orientation|screenSize" 
            android:theme="@style/LaunchScreenTheme"
            android:screenOrientation="portrait">

            <meta-data android:name="SET_THEME_ON_LAUNCH" android:resource="@style/AppTheme" /> 

            <intent-filter> 
            <action android:name="android.intent.action.MAIN" /> 
            <category android:name="android.intent.category.LAUNCHER" /> 
            </intent-filter> 
        </activity>
        <activity android:name="com.tns.ErrorReportActivity"/>
    </application>
</manifest>

app.gradle:

android {  
  defaultConfig {  
    multiDexEnabled true
    generatedDensities = []
    ndk {
       abiFilters.clear()
       abiFilters.addAll(['armeabi-v7a','arm64-v8a'])
   }
  }  
  aaptOptions {  
    additionalParameters "--no-version-vectors"  
  }  
  dexOptions {
    javaMaxHeapSize "4g"
  }
} 

This is a big issue for us, we need to be able to scan 200-300 times a row with pauses of 2-3 seconds with the pause variable. You help us a lot with your plugins thank you for everything and your future help !

[kriefsacha]

@EddyVerbruggen can you help please ? I have the same issue here..

[kriefsacha]

@NathanWalker Can you help maybe please ? It's a big issue ..

[neeraj-algoworks]

@EinatGit @kriefsacha Did you find the solution?

[kriefsacha]

@EinatGit @kriefsacha Did you find the solution?

No.. no one is responding.. you have the same issue ?

[neeraj-algoworks]

@EddyVerbruggen @NathanWalker @KkevinLi @tbozhikov @vcooley @sean-perkins Can you please help. I am facing the same issue for Android devices. Although its working fine for iOS devices.

[neeraj-algoworks]

@EinatGit @kriefsacha Did you find the solution?

No.. no one is responding.. you have the same issue ?

Yes , I am facing the same

[NathanWalker]

Just seeing this for first time - had been focused on a lot of {N} 7 details - we’ll bump this up in priority as it sounds like a Java/android adjustment could be made to help there - @triniwiz may have some thoughts as well. Best case would be mid next week as we’d love to help with more on this plugin in general.

[kriefsacha]

Thank you a lot @NathanWalker ! Indeed it's a plugin that i use in every project, you did an awesome work here. But this is a big issue for us, so thanks !

Just to say, try also on IOS , i had the same issue on Iphone 7.

[kriefsacha]

Hello there @NathanWalker ! Do you have some news about this ?

[DanielDent]

I believe there is a long-standing iOS-specific memory leak that I don't think was ever fully resolved:

https://github.com/EddyVerbruggen/nativescript-plugin-firebase/issues/1190

The way to get the crash is to open the barcode scanner and simply leave it running. While #1190 greatly improved things, I think it's still the case that frames can end up in a processing queue of unbounded size. Google recommends: "Throttle calls to the detector. If a new video frame becomes available while the detector is running, drop the frame."

[triniwiz]

Thanks for the info , I’ll be looking into this very soon 

Sent from Yahoo Mail for iPhone

On Sunday, September 27, 2020, 11:32 PM, Daniel Dent notifications@github.com wrote:

I believe there is a long-standing iOS-specific memory leak that I don't think was ever fully resolved:

1190

The way to get the crash is to open the barcode scanner and simply leave it running. While #1190 greatly improved things, I think it's still the case that frames can end up in a processing queue of unbounded size. Google recommends: "Throttle calls to the detector. If a new video frame becomes available while the detector is running, drop the frame."

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[kriefsacha]

Hi @triniwiz @NathanWalker , some news about it ?

[triniwiz]

I’ll try to take a look this week 

Sent from Yahoo Mail for iPhone

On Sunday, October 11, 2020, 3:19 AM, kriefsacha notifications@github.com wrote:

Hi @triniwiz @NathanWalker , some news about it ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

[kriefsacha]

I'm sorry to ask again and again but that's a very big and important issue for us. Do you have news @triniwiz @NathanWalker ?

[EinatGit]

Is there any news about this issue? @triniwiz @NathanWalker

[kriefsacha]

Okay so @triniwiz @NathanWalker , we really had to progress about this issue so we did some debug of the code. And we found some things that can help.

• On the file mlkit/mlkit-cameraview.android.ts, we saw that the resolution of the pictureSize is too big. it was around 3200/4600. So we tried to put the same resolution of the preview by doing line 156:

        const pictureSize = sizePair.previewSize;
        const previewSize = sizePair.previewSize;

And then line 149 instead of 1400,1200 we puted 700,600.

With this, we achieved from around 30-50 scans in a row before to around 200 !

We think by doing this garbage collector had more time to do his job.

• On the same file line 386 there is the function

createPreviewBuffer

By doing some console logs we saw that one of the two errors (i will speak about the second one after) it's from here. Because it's called on try and catch the app doesnt crash but it freeze. But we didn't managed to see why this function was doing an error and where in the function.

• On the file mlkit/barcodescanning/index.android.ts line 62 there is the function

createSuccessListener

on this function line 72 you take the data of the last vision to after that just returning the width and height. The second error (the one we had from the beginning you can see on the stack trace:

System.err: at com.google.firebase.ml.vision.common.FirebaseVisionImage.getBitmap(Unknown Source:46)

) is from this. There is no try and catch there so that's why the app was crashing. We puted in comment this part, we wasn't returning the image anymore and then we didn't have the error. We had just the first error of the buffer.

We actually didn't understand why do someone need to receive the height and width of the result, and even so, maybe make it optional or something.

So that's what we found. One way to make the error coming later, and two places where the error can come from. If it can help and we would be glad to have your opinion on all this.

[triniwiz]

Thanks for the update I’m currently writing something to handle things better in my initial test (image labeling / object detection really can’t remember the name 🙈) I had like 500+ detections w/o crashing

[kriefsacha]

That's great @triniwiz !

Just keep in mind what i puted here maybe it can help.

And do you know around how much time it will be available ?

[kriefsacha]

Hi @triniwiz .

Another time, this is a big issue for us. We are offering our help if you need it on the issue.

We also wanted to know if you know even with your optimisations, why after 500 at some point it crashes ?

Thank you for your help.

[kriefsacha]

Hi @triniwiz @NathanWalker we don't have any news for a while now...

Are you working on this please ?

Thank you