search

bytedeco / javacv

Java interface to OpenCV, FFmpeg, and more
Other
7.51k stars 1.58k forks source link

JavaCV in Kotlin - Memory Allocation/Management? #1975

Open gingerbraden opened 1 year ago

gingerbraden commented 1 year ago

I'm building an app in Kotlin that heavily depends on image processing through saliency.

My code looks like this:

package com.example.testingsaliency5

import android.graphics.Bitmap
import android.os.Bundle
import androidx.appcompat.app.AppCompatActivity
import androidx.core.graphics.drawable.toBitmap
import com.example.testingsaliency5.databinding.ActivityMainBinding
import org.bytedeco.javacv.AndroidFrameConverter
import org.bytedeco.javacv.OpenCVFrameConverter.ToMat
import org.bytedeco.opencv.opencv_core.Mat
import org.bytedeco.opencv.opencv_saliency.StaticSaliencyFineGrained

class MainActivity : AppCompatActivity() {
    private lateinit var binding: ActivityMainBinding

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        binding = ActivityMainBinding.inflate(layoutInflater)
        val view = binding.root
        binding.imageView.setImageDrawable(this.resources.getDrawable(R.drawable.lancer))
        binding.button.setOnClickListener { binding.imageView.setImageBitmap(saliencyBitmap(binding.imageView.drawable.toBitmap())) }
        setContentView(view)
    }

    fun saliencyBitmap(bitmap : Bitmap) : Bitmap {

        val converterToBitmap = AndroidFrameConverter()
        val converterToMat = ToMat()

        val frame = converterToBitmap.convert(bitmap)
        val saliencyMap = Mat()
        val byteMap = Mat()
        val sal = StaticSaliencyFineGrained()
        sal.computeSaliency(converterToMat.convert(frame), saliencyMap)
        sal.computeBinaryMap(saliencyMap, byteMap)

        return converterToBitmap.convert(converterToMat.convert(byteMap))
    }
}

But when ran, after I press the button, I get this error on line return converterToBitmap.convert(converterToMat.convert(byteMap)):

I/estingsaliency5: Starting a blocking GC Alloc
I/estingsaliency5: Starting a blocking GC Alloc
I/estingsaliency5: Alloc young concurrent copying GC freed 190867(12MB) AllocSpace objects, 0(0B) LOS objects, 86% free, 3863KB/27MB, paused 29us,12us total 22.231ms
I/estingsaliency5: WaitForGcToComplete blocked NativeAlloc on Alloc for 22.376ms
I/estingsaliency5: Forcing collection of SoftReferences for 254MB allocation
I/estingsaliency5: Starting a blocking GC Alloc
I/estingsaliency5: Alloc concurrent copying GC freed 1873(61KB) AllocSpace objects, 0(0B) LOS objects, 86% free, 3817KB/27MB, paused 20us,13us total 9.242ms
W/estingsaliency5: Throwing OutOfMemoryError "Failed to allocate a 266501312 byte allocation with 25165824 free bytes and 252MB until OOM, target footprint 29075352, growth limit 268435456" (VmSize 16893380 kB)
I/estingsaliency5: Starting a blocking GC Alloc
I/estingsaliency5: Starting a blocking GC Alloc
I/estingsaliency5: Forcing collection of SoftReferences for 254MB allocation
I/estingsaliency5: Starting a blocking GC Alloc
I/estingsaliency5: Alloc concurrent copying GC freed 1774(71KB) AllocSpace objects, 0(0B) LOS objects, 86% free, 3746KB/27MB, paused 17us,14us total 8.149ms
W/estingsaliency5: Throwing OutOfMemoryError "Failed to allocate a 266501312 byte allocation with 25165824 free bytes and 252MB until OOM, target footprint 29002392, growth limit 268435456" (VmSize 16893380 kB)
D/AndroidRuntime: Shutting down VM
E/AndroidRuntime: FATAL EXCEPTION: main
    Process: com.example.testingsaliency5, PID: 4452
    java.lang.OutOfMemoryError: Failed to allocate a 266501312 byte allocation with 25165824 free bytes and 252MB until OOM, target footprint 29002392, growth limit 268435456
        at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:54)
        at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:49)
        at java.nio.ByteBuffer.allocate(ByteBuffer.java:284)
        at org.bytedeco.javacv.AndroidFrameConverter.gray2rgba(AndroidFrameConverter.java:131)
        at org.bytedeco.javacv.AndroidFrameConverter.convert(AndroidFrameConverter.java:201)
        at com.example.testingsaliency5.MainActivity.saliencyBitmap(MainActivity.kt:38)
        at com.example.testingsaliency5.MainActivity.onCreate$lambda$0(MainActivity.kt:22)
        at com.example.testingsaliency5.MainActivity.$r8$lambda$71u2Lpx-hlCZ4uuH3x2VT4aZAZY(Unknown Source:0)
        at com.example.testingsaliency5.MainActivity$$ExternalSyntheticLambda0.onClick(Unknown Source:2)
        at android.view.View.performClick(View.java:7506)
        at com.google.android.material.button.MaterialButton.performClick(MaterialButton.java:1219)
        at android.view.View.performClickInternal(View.java:7483)
        at android.view.View.-$$Nest$mperformClickInternal(Unknown Source:0)
        at android.view.View$PerformClick.run(View.java:29341)
        at android.os.Handler.handleCallback(Handler.java:942)
        at android.os.Handler.dispatchMessage(Handler.java:99)
        at android.os.Looper.loopOnce(Looper.java:201)
        at android.os.Looper.loop(Looper.java:288)
        at android.app.ActivityThread.main(ActivityThread.java:7872)
        at java.lang.reflect.Method.invoke(Native Method)
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:548)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:936)
I/Process: Sending signal. PID: 4452 SIG: 9

The quesiton is, do I need to manage memory in some way, when working with JavaCV on Android?

saudet commented 1 year ago

It's not necessary, but it's recommended with PointerScope yes: http://bytedeco.org/news/2018/07/17/bytedeco-as-distribution/

AndroidFrameConverter.gray2rgba() should allocate memory through JavaCPP to avoid that error though... Contributions are welcome!

gingerbraden commented 1 year ago

After wrapping the function like this:

PointerScope().use { scope -> return converterToBitmap.convert(converterToMat.convert(byteMap)) }

I am also getting these lines of errors:

E/cv::error(): OpenCV(4.6.0) Error: Requested object was not found (could not open directory: /data/app/~~mkI7ZXIjkcXrkLe2ozCALg==/com.example.testingsaliency5-xD2aRrfEAJXDI_AdGJeCoA==/base.apk!/lib/arm64-v8a) in glob_rec, file /__w/javacpp-presets/javacpp-presets/opencv/cppbuild/android-arm64/opencv-4.6.0/modules/core/src/glob.cpp, line 279
E/linker: normalize_path - invalid input: "lib/", the input path should be absolute
W/linker: Warning: unable to normalize "lib/" (ignoring)
E/linker: normalize_path - invalid input: "lib/", the input path should be absolute
W/linker: Warning: unable to normalize "lib/" (ignoring)
W/System.err: Warning: Version of org.bytedeco:openblas could not be found.
W/System.err: Warning: Version of org.bytedeco:opencv could not be found.

with many repeating copies of 

E/BLASTBufferQueue: [VRI[MainActivity]#0](f:0,a:3) Faking releaseBufferCallback from transactionCompleteCallback

and finally the app crashes on the same error as above.

saudet commented 1 year ago

We have to use PointerScope where memory gets allocated with JavaCPP. I told you the problem here is that AndroidFrameConverter.gray2rgba() does not currently use JavaCPP to allocate memory, so PointerScope cannot have any effect there, that's normal.

gingerbraden commented 1 year ago

Oh, sorry I understood

AndroidFrameConverter.gray2rgba() should allocate memory through JavaCPP to avoid that error though...

as that gray2rgba does allocate memory

saudet commented 1 year ago

Try to use PointerScope in other places where you use JavaCV. That should work around the issue at least.

gingerbraden commented 1 year ago

I added PointerScope to all uses of JavaCV like so:

val converterToBitmap = AndroidFrameConverter()
        val converterToMat = ToMat()
        var frame = Frame();
        PointerScope().use { scope -> frame = converterToBitmap.convert(bitmap) }
        Log.d("saliency", "1")
        val saliencyMap = Mat()
        val byteMap = Mat()
        val sal = StaticSaliencyFineGrained()
        PointerScope().use { scope -> sal.computeSaliency(converterToMat.convert(frame), saliencyMap) }
        Log.d("saliency", "2")
        PointerScope().use { scope -> sal.computeBinaryMap(saliencyMap, byteMap) }
        Log.d("saliency", "3")
        PointerScope().use { scope -> return converterToBitmap.convert(converterToMat.convert(byteMap)) }

And it didn't help. I managed to get this funny error:

CPU usage from 0ms to 9543ms later (2023-01-23 10:35:02.363 to 2023-01-23 10:35:11.906) with 99% awake:
177% 12913/com.example.testingsaliency5: 176% user + 0.7% kernel / faults: 4922 minor 77 major

But this is the usueal log since start of the app to its crash. Interesting part is that can't find some objects in javacpp-presets, even though I have them in my dependencies.

2023-01-23 10:54:23.369 20734-20754 AdrenoUtils             com.example.testingsaliency5         W  <ReadGpuID_from_sysfs:197>: Failed to open /sys/class/kgsl/kgsl-3d0/gpu_model
2023-01-23 10:54:23.369 20734-20754 AdrenoUtils             com.example.testingsaliency5         W  <ReadGpuID:221>: Failed to read chip ID from gpu_model. Fallback to use the GSL path
2023-01-23 10:54:23.380 20734-20754 OpenGLRenderer          com.example.testingsaliency5         E  Unable to match the desired swap behavior.
2023-01-23 10:54:23.415 20734-20754 Parcel                  com.example.testingsaliency5         W  Expecting binder but got null!
2023-01-23 10:54:29.842 20734-20746 System                  com.example.testingsaliency5         W  A resource failed to call close. 
2023-01-23 10:54:30.894 20734-20734 linker                  com.example.testingsaliency5         E  normalize_path - invalid input: "lib/", the input path should be absolute
2023-01-23 10:54:30.894 20734-20734 linker                  com.example.testingsaliency5         W  Warning: unable to normalize "lib/" (ignoring)
2023-01-23 10:54:30.894 20734-20734 linker                  com.example.testingsaliency5         E  normalize_path - invalid input: "lib/", the input path should be absolute
2023-01-23 10:54:30.894 20734-20734 linker                  com.example.testingsaliency5         W  Warning: unable to normalize "lib/" (ignoring)
2023-01-23 10:54:30.908 20734-20734 System.err              com.example.testingsaliency5         W  Warning: Version of org.bytedeco:openblas could not be found.
2023-01-23 10:54:30.911 20734-20734 System.err              com.example.testingsaliency5         W  Warning: Version of org.bytedeco:opencv could not be found.
2023-01-23 10:54:31.397 20734-20734 saliency                com.example.testingsaliency5         D  1
2023-01-23 10:54:32.109 20734-20734 cv::error()             com.example.testingsaliency5         E  OpenCV(4.6.0) Error: Requested object was not found (could not open directory: /data/app/~~mkI7ZXIjkcXrkLe2ozCALg==/com.example.testingsaliency5-xD2aRrfEAJXDI_AdGJeCoA==/base.apk!/lib/arm64-v8a) in glob_rec, file /__w/javacpp-presets/javacpp-presets/opencv/cppbuild/android-arm64/opencv-4.6.0/modules/core/src/glob.cpp, line 279
2023-01-23 10:54:32.130 20734-20734 cv::error()             com.example.testingsaliency5         E  OpenCV(4.6.0) Error: Requested object was not found (could not open directory: /data/app/~~mkI7ZXIjkcXrkLe2ozCALg==/com.example.testingsaliency5-xD2aRrfEAJXDI_AdGJeCoA==/base.apk!/lib/arm64-v8a) in glob_rec, file /__w/javacpp-presets/javacpp-presets/opencv/cppbuild/android-arm64/opencv-4.6.0/modules/core/src/glob.cpp, line 279
2023-01-23 10:54:32.131 20734-20734 cv::error()             com.example.testingsaliency5         E  OpenCV(4.6.0) Error: Requested object was not found (could not open directory: /data/app/~~mkI7ZXIjkcXrkLe2ozCALg==/com.example.testingsaliency5-xD2aRrfEAJXDI_AdGJeCoA==/base.apk!/lib/arm64-v8a) in glob_rec, file /__w/javacpp-presets/javacpp-presets/opencv/cppbuild/android-arm64/opencv-4.6.0/modules/core/src/glob.cpp, line 279
2023-01-23 10:54:52.409 20734-20734 saliency                com.example.testingsaliency5         D  2
2023-01-23 10:55:19.763 20734-20734 saliency                com.example.testingsaliency5         D  3
2023-01-23 10:55:19.778 20734-20734 estingsaliency5         com.example.testingsaliency5         I  Starting a blocking GC Alloc
2023-01-23 10:55:19.778 20734-20734 estingsaliency5         com.example.testingsaliency5         I  Starting a blocking GC Alloc
2023-01-23 10:55:19.797 20734-20734 estingsaliency5         com.example.testingsaliency5         I  Alloc young concurrent copying GC freed 192328(12MB) AllocSpace objects, 0(0B) LOS objects, 86% free, 3833KB/27MB, paused 28us,28us total 18.992ms
2023-01-23 10:55:19.797 20734-20734 estingsaliency5         com.example.testingsaliency5         I  Forcing collection of SoftReferences for 254MB allocation
2023-01-23 10:55:19.797 20734-20744 estingsaliency5         com.example.testingsaliency5         I  WaitForGcToComplete blocked NativeAlloc on Alloc for 19.206ms
2023-01-23 10:55:19.797 20734-20734 estingsaliency5         com.example.testingsaliency5         I  Starting a blocking GC Alloc
2023-01-23 10:55:19.808 20734-20734 estingsaliency5         com.example.testingsaliency5         I  Alloc concurrent copying GC freed 1859(61KB) AllocSpace objects, 0(0B) LOS objects, 86% free, 3772KB/27MB, paused 21us,894us total 10.880ms
2023-01-23 10:55:19.809 20734-20734 estingsaliency5         com.example.testingsaliency5         W  Throwing OutOfMemoryError "Failed to allocate a 266501312 byte allocation with 25165824 free bytes and 252MB until OOM, target footprint 29028528, growth limit 268435456" (VmSize 17062520 kB)
2023-01-23 10:55:19.809 20734-20734 estingsaliency5         com.example.testingsaliency5         I  Starting a blocking GC Alloc
2023-01-23 10:55:19.809 20734-20734 estingsaliency5         com.example.testingsaliency5         I  Starting a blocking GC Alloc
2023-01-23 10:55:19.813 20734-20734 estingsaliency5         com.example.testingsaliency5         I  Forcing collection of SoftReferences for 254MB allocation
2023-01-23 10:55:19.813 20734-20734 estingsaliency5         com.example.testingsaliency5         I  Starting a blocking GC Alloc
2023-01-23 10:55:19.821 20734-20734 estingsaliency5         com.example.testingsaliency5         I  Alloc concurrent copying GC freed 2037(79KB) AllocSpace objects, 0(0B) LOS objects, 86% free, 3708KB/27MB, paused 14us,15us total 7.827ms
2023-01-23 10:55:19.821 20734-20734 estingsaliency5         com.example.testingsaliency5         W  Throwing OutOfMemoryError "Failed to allocate a 266501312 byte allocation with 25165824 free bytes and 252MB until OOM, target footprint 28963440, growth limit 268435456" (VmSize 17062520 kB)
2023-01-23 10:55:19.821 20734-20734 AndroidRuntime          com.example.testingsaliency5         D  Shutting down VM
2023-01-23 10:55:19.835 20734-20734 AndroidRuntime          com.example.testingsaliency5         E  FATAL EXCEPTION: main
                                                                                                    Process: com.example.testingsaliency5, PID: 20734
                                                                                                    java.lang.OutOfMemoryError: Failed to allocate a 266501312 byte allocation with 25165824 free bytes and 252MB until OOM, target footprint 28963440, growth limit 268435456
saudet commented 1 year ago

If that code for converterToBitmap is the only place where you're using JavaCV, try to use a single PointerScope around all that.