Knift has a memory leak in Android

[hbyjw]

I use AAR files to develop applications, and there is a memory leak. Then I tested that the APK generated using the script in the document will still leak memory.

System information (Please provide as much relevant information as possible)

• custom code:

  1. mediapipe/java/com/google/mediapipe/mediapipe_aar.bzl replace libopencv_java3 with libopencv_java4
  2. mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic/AndroidManifest.xml -> applaciton add android:debuggable="true"

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04, Android 11, iOS 14.4):

  1. Linux Ubuntu 20.04 ,Android 9

• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:

  1. Samsung Galaxy Note8

• Programming Language and version ( e.g. C++, Python, Java):

  1. Java

• MediaPipe version: version:The latest code on GitHub

• Bazel version (if compiling from source):

• Solution ( e.g. FaceMesh, Pose, Holistic ):

  1. KNIFT

Describe the current behavior:

1. Use the following command to generate the APK file.

   
   # Switch to OpenCV 4
   sed -i -e 's:3.4.3/opencv-3.4.3:4.0.1/opencv-4.0.1:g' WORKSPACE
   sed -i -e 's:libopencv_java3:libopencv_java4:g' third_party/opencv_android.BUILD

Build and install app

bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/templatematchingcpu

Switch back to OpenCV 3

sed -i -e 's:4.0.1/opencv-4.0.1:3.4.3/opencv-3.4.3:g' WORKSPACE sed -i -e 's:libopencv_java4:libopencv_java3:g' third_party/opencv_android.BUILD

2. Install APK, and then use the profiler of Android studio to detect memory changes. The mobile phone is placed on the desktop without moving, and there is no object to be recognized in the camera viewfinder box.

https://user-images.githubusercontent.com/13883357/167331161-eb4cd0fe-ffbc-449c-9cda-072452f3db66.MOV

3.  Run out of memory until the application crashes.
![截屏2022-05-09 10 54 17](https://user-images.githubusercontent.com/13883357/167332890-a8f70a59-4a30-419e-ae65-95cdad2e8f18.png)

**Describe the expected behavior:**

**Standalone code to reproduce the issue:**

**Other info / Complete Logs :**
[knift.log](https://github.com/google/mediapipe/files/8648506/knift.log)

[sureshdagooglecom]

Hi @hbyjw , Did you get a chance to closed issue https://github.com/google/mediapipe/issues/1089

[hbyjw]

@sureshdagooglecom

I'll test the method used #1089 tomorrow

[hbyjw]

@sureshdagooglecom

I modified the code according to #1089 information, but there will still be memory leakage. When a value is reached, the camera preview interface will be stuck in the last position

https://user-images.githubusercontent.com/13883357/167572895-ce3fe1ac-0de6-4a2f-a613-441f412a3072.MOV

[hbyjw]

@sureshdagooglecom

不知道下面的图片对解决问题是否有帮助，这是使用的Android Studio 中的Profiler Record Natvie allocations

I don't know whether the following pictures are helpful to solve the problem. This is the profiler record natvie allocations in Android studio

[hbyjw]

@sureshdagooglecom

I added some code

mediapipe/calculators/image/feature_detector_calculator.cc -> absl::Status FeatureDetectorCalculator::Process(CalculatorContext* cc) add free(tensor.data.data);// 释放内存

I don't know whether the added code is appropriate. My C + + level is very low. The added results basically meet my needs.

[sureshdagooglecom]

Hi @hbyjw , Are you still facing any issues w.r.t above issue?

[yegenzhi]

hi, @hbyjw i guess the problem is: you list "PATCHES" as an output stream for feature_detector_calculator but that stream is not consumed by any downstream calculator. so if you don't need that PATCHES stream, just try delete it in the graph. let me know if that works.

[hbyjw]

@yegenzhi

不知道我的操作是否符合你的意思，见图片。这样操作生成的aar以及 mobile_cpu.binarypb 会出现如下错误。 因为要获取标记框的坐标，我参考https://github.com/google/mediapipe/issues/1624#issuecomment-782265180 在labeled_detections中添加了TimedBoxProtoList的内容

English is not good. The following content is translated by software

I don't know whether my operation is in line with your meaning. See the picture. New AAR and mobile_cpu.binarypb has the following error. I need to get the coordinates of the marker box, so refer to https://github.com/google/mediapipe/issues/1624#issuecomment -782265180 added TimedBoxProtoList to labeled_detections

2022-05-20 11:18:42.251 20734-20842/com.yjw.yujiaweitestknift E/native: E20220520 11:18:42.251859 20842 graph.cc:472] ; Input Stream "patches" for node with sorted index 4 does not have a corresponding output stream.
2022-05-20 11:18:42.256 20734-20842/com.yjw.yujiaweitestknift E/AndroidRuntime: FATAL EXCEPTION: ExternalTextureConverter
    Process: com.yjw.yujiaweitestknift, PID: 20734
    com.google.mediapipe.framework.MediaPipeException: unknown: ; Input Stream "patches" for node with sorted index 4 does not have a corresponding output stream.
        at com.google.mediapipe.framework.Graph.nativeStartRunningGraph(Native Method)
        at com.google.mediapipe.framework.Graph.startRunningGraph(Graph.java:324)
        at com.google.mediapipe.components.FrameProcessor.startGraph(FrameProcessor.java:553)
        at com.google.mediapipe.components.FrameProcessor.maybeAcceptNewFrame(FrameProcessor.java:421)
        at com.google.mediapipe.components.FrameProcessor.onNewFrame(FrameProcessor.java:439)
        at com.google.mediapipe.components.ExternalTextureConverter$RenderThread.renderNext(ExternalTextureConverter.java:403)
        at com.google.mediapipe.components.ExternalTextureConverter$RenderThread.lambda$onFrameAvailable$0$ExternalTextureConverter$RenderThread(ExternalTextureConverter.java:338)
        at com.google.mediapipe.components.-$$Lambda$ExternalTextureConverter$RenderThread$IXJjtE68JMV2CMJNQcFYhTUqhj8.run(Unknown Source:4)
        at android.os.Handler.handleCallback(Handler.java:873)
        at android.os.Handler.dispatchMessage(Handler.java:99)
        at android.os.Looper.loop(Looper.java:214)
        at com.google.mediapipe.glutil.GlThread.run(GlThread.java:141)

[sureshdagooglecom]

Hi @hbyjw , This appears to be a leak in native libraries specific to this Knift demo.
We will investigate further when we can.

[hbyjw]

@sureshdagooglecom OK.

[ayushgdev]

Hello @hbyjw We are ending support for these MediaPipe Legacy Solutions, but upgrading the others. However, the libraries, documentation, and source code for all the MediapPipe Legacy Solutions will continue to be available in our GitHub repository and through library distribution services, such as Maven and NPM.

You can continue to use those legacy solutions in your applications if you choose. Though, we would request you to check new MediaPipe solutions which can help you more easily build and customize ML solutions for your applications. These new solutions will provide a superset of capabilities available in the legacy solutions.

[google-ml-butler[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

[github-actions[bot]]

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

[github-actions[bot]]

This issue was closed due to lack of activity after being marked stale for past 7 days.

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No