search

google-ai-edge / mediapipe

Cross-platform, customizable ML solutions for live and streaming media.
https://ai.google.dev/edge/mediapipe
Apache License 2.0
27.41k stars 5.15k forks source link

[adnroid ] unable to get camera preview when using objectdectioncpu, objectdectiongpu works. #1784

Closed yangcheng closed 3 years ago

yangcheng commented 3 years ago

The same device can run mlkit's sample with no issue https://github.com/googlesamples/mlkit

however, when running mediapipe's objectdectioncpu example, I can't get a camera preview or any results.

objectdectiongpu can open the camera with a longer delay than mlkit. where should I start to debug the problem? Thanks

Here is the log I run objectdectioncpu 

2021-03-22 14:46:53.049 5736-5736/com.google.mediapipe.apps.objectdetectioncpu W/ActivityThread: handleWindowVisibility: no activity for token android.os.BinderProxy@c9b8066
2021-03-22 14:46:53.089 5736-5736/com.google.mediapipe.apps.objectdetectioncpu I/native: I20210322 14:46:53.089723  5736 asset_manager_util.cc:59] Created global reference to asset manager.
2021-03-22 14:46:53.095 5736-5736/com.google.mediapipe.apps.objectdetectioncpu E/libEGL: eglMakeCurrent:1028 error 300d (EGL_BAD_SURFACE)
2021-03-22 14:46:53.095 5736-5736/com.google.mediapipe.apps.objectdetectioncpu I/native: I20210322 14:46:53.095924  5736 gl_context_egl.cc:163] Successfully initialized EGL. Major : 1 Minor: 4
2021-03-22 14:46:53.097 5736-5836/com.google.mediapipe.apps.objectdetectioncpu I/native: I20210322 14:46:53.097872  5836 gl_context.cc:331] GL version: 3.2 (OpenGL ES 3.2 v1.r16p0-01rel0.45b161e1b2cff94b03faf73ffeb6b9a5)
2021-03-22 14:46:53.098 5736-5736/com.google.mediapipe.apps.objectdetectioncpu D/PermissionHelper: checkAndRequestCameraPermissions
2021-03-22 14:46:53.104 5736-5837/com.google.mediapipe.apps.objectdetectioncpu D/GlThread: Starting GL thread ExternalTextureConverter
2021-03-22 14:46:53.128 5736-5736/com.google.mediapipe.apps.objectdetectioncpu D/CameraOrientationUtil: getRelativeImageRotation: destRotationDegrees=0, sourceRotationDegrees=0, isOppositeFacing=true, result=0
2021-03-22 14:46:53.132 5736-5736/com.google.mediapipe.apps.objectdetectioncpu D/DeferrableSurface: Surface created[total_surfaces=1, used_surfaces=0](androidx.camera.core.SurfaceRequest$2@76a864c}
2021-03-22 14:46:53.135 5736-5736/com.google.mediapipe.apps.objectdetectioncpu D/CameraOrientationUtil: getRelativeImageRotation: destRotationDegrees=0, sourceRotationDegrees=0, isOppositeFacing=true, result=0
2021-03-22 14:46:53.135 5736-5838/com.google.mediapipe.apps.objectdetectioncpu D/CameraXPreviewHelper: Received surface request for resolution 1600x1200
2021-03-22 14:46:53.142 5736-5777/com.google.mediapipe.apps.objectdetectioncpu D/UseCaseAttachState: Active and attached use case: [] for camera: 0
2021-03-22 14:46:53.144 5736-5777/com.google.mediapipe.apps.objectdetectioncpu D/UseCaseAttachState: All use case: [androidx.camera.core.Preview-fecf4293-85a3-4c2d-86f6-4526634ebb08161899993] for camera: 0
2021-03-22 14:46:53.145 5736-5777/com.google.mediapipe.apps.objectdetectioncpu D/UseCaseAttachState: Active and attached use case: [androidx.camera.core.Preview-fecf4293-85a3-4c2d-86f6-4526634ebb08161899993] for camera: 0
2021-03-22 14:46:53.146 5736-5838/com.google.mediapipe.apps.objectdetectioncpu I/ectdetectioncp: IncrementDisableThreadFlip blocked for 11.112ms
2021-03-22 14:46:53.148 5736-5777/com.google.mediapipe.apps.objectdetectioncpu D/CameraStateRegistry: tryOpenCamera(Camera@9fbaa47[id=0]) [Available Cameras: 1, Already Open: false (Previous state: CLOSED)] --> SUCCESS
2021-03-22 14:46:53.149 5736-5777/com.google.mediapipe.apps.objectdetectioncpu D/CameraStateRegistry: Recalculating open cameras:
    Camera                                       State                 
    -------------------------------------------------------------------
    Camera@9fbaa47[id=0]                         OPENING               
    Camera@a2d21e0[id=1]                         UNKNOWN               
    -------------------------------------------------------------------
    Open count: 1 (Max allowed: 1)
2021-03-22 14:46:53.149 5736-5777/com.google.mediapipe.apps.objectdetectioncpu D/UseCaseAttachState: All use case: [androidx.camera.core.Preview-fecf4293-85a3-4c2d-86f6-4526634ebb08161899993] for camera: 0
2021-03-22 14:46:53.150 5736-5838/com.google.mediapipe.apps.objectdetectioncpu D/CameraXPreviewHelper: Providing surface
2021-03-22 14:46:53.151 5736-5778/com.google.mediapipe.apps.objectdetectioncpu D/mali_winsys: EGLint new_window_surface(egl_winsys_display *, void *, EGLSurface, EGLConfig, egl_winsys_surface **, EGLBoolean) returns 0x3000
2021-03-22 14:46:53.172 5736-5777/com.google.mediapipe.apps.objectdetectioncpu D/UseCaseAttachState: Active and attached use case: [androidx.camera.core.Preview-fecf4293-85a3-4c2d-86f6-4526634ebb08161899993] for camera: 0
2021-03-22 14:46:53.175 5736-5777/com.google.mediapipe.apps.objectdetectioncpu D/CameraStateRegistry: Recalculating open cameras:
    Camera                                       State                 
    -------------------------------------------------------------------
    Camera@9fbaa47[id=0]                         OPEN                  
    Camera@a2d21e0[id=1]                         UNKNOWN               
    -------------------------------------------------------------------
    Open count: 1 (Max allowed: 1)
2021-03-22 14:46:53.176 5736-5777/com.google.mediapipe.apps.objectdetectioncpu D/UseCaseAttachState: All use case: [androidx.camera.core.Preview-fecf4293-85a3-4c2d-86f6-4526634ebb08161899993] for camera: 0
2021-03-22 14:46:53.178 5736-5777/com.google.mediapipe.apps.objectdetectioncpu D/DeferrableSurface: New surface in use[total_surfaces=1, used_surfaces=1](androidx.camera.core.SurfaceRequest$2@76a864c}
2021-03-22 14:46:53.178 5736-5777/com.google.mediapipe.apps.objectdetectioncpu D/DeferrableSurface: use count+1, useCount=1 androidx.camera.core.SurfaceRequest$2@76a864c
2021-03-22 14:46:53.178 5736-5777/com.google.mediapipe.apps.objectdetectioncpu D/CaptureSession: Opening capture session.
2021-03-22 14:46:53.284 5736-5777/com.google.mediapipe.apps.objectdetectioncpu D/CaptureSession: Attempting to send capture request onConfigured
2021-03-22 14:46:53.284 5736-5777/com.google.mediapipe.apps.objectdetectioncpu D/CaptureSession: Issuing request for session.
2021-03-22 14:46:53.289 5736-5777/com.google.mediapipe.apps.objectdetectioncpu D/CaptureSession: CameraCaptureSession.onConfigured() mState=OPENED
2021-03-22 14:46:53.289 5736-5777/com.google.mediapipe.apps.objectdetectioncpu D/CaptureSession: CameraCaptureSession.onReady() OPENED
2021-03-22 14:46:53.620 5736-5838/com.google.mediapipe.apps.objectdetectioncpu D/CameraOrientationUtil: getRelativeImageRotation: destRotationDegrees=0, sourceRotationDegrees=0, isOppositeFacing=true, result=0
2021-03-22 14:46:53.653 5736-5836/com.google.mediapipe.apps.objectdetectioncpu D/mali_winsys: EGLint new_window_surface(egl_winsys_display *, void *, EGLSurface, EGLConfig, egl_winsys_surface **, EGLBoolean) returns 0x3000

and logs for objectdectiongpu 

2021-03-22 14:48:38.680 5668-5668/com.google.mediapipe.apps.objectdetectiongpu W/ActivityThread: handleWindowVisibility: no activity for token android.os.BinderProxy@264f0c1
2021-03-22 14:48:38.728 5668-5668/com.google.mediapipe.apps.objectdetectiongpu I/native: I20210322 14:48:38.728003  5668 asset_manager_util.cc:59] Created global reference to asset manager.
2021-03-22 14:48:38.730 5668-5668/com.google.mediapipe.apps.objectdetectiongpu E/libEGL: eglMakeCurrent:1028 error 300d (EGL_BAD_SURFACE)
2021-03-22 14:48:38.731 5668-5668/com.google.mediapipe.apps.objectdetectiongpu I/native: I20210322 14:48:38.731772  5668 gl_context_egl.cc:163] Successfully initialized EGL. Major : 1 Minor: 4
2021-03-22 14:48:38.734 5668-5853/com.google.mediapipe.apps.objectdetectiongpu I/native: I20210322 14:48:38.734324  5853 gl_context.cc:331] GL version: 3.2 (OpenGL ES 3.2 v1.r16p0-01rel0.45b161e1b2cff94b03faf73ffeb6b9a5)
2021-03-22 14:48:38.735 5668-5668/com.google.mediapipe.apps.objectdetectiongpu D/PermissionHelper: checkAndRequestCameraPermissions
2021-03-22 14:48:38.747 5668-5854/com.google.mediapipe.apps.objectdetectiongpu D/GlThread: Starting GL thread ExternalTextureConverter
2021-03-22 14:48:38.770 5668-5668/com.google.mediapipe.apps.objectdetectiongpu D/CameraOrientationUtil: getRelativeImageRotation: destRotationDegrees=0, sourceRotationDegrees=0, isOppositeFacing=true, result=0
2021-03-22 14:48:38.774 5668-5668/com.google.mediapipe.apps.objectdetectiongpu D/DeferrableSurface: Surface created[total_surfaces=1, used_surfaces=0](androidx.camera.core.SurfaceRequest$2@7b2667f}
2021-03-22 14:48:38.775 5668-5668/com.google.mediapipe.apps.objectdetectiongpu D/CameraOrientationUtil: getRelativeImageRotation: destRotationDegrees=0, sourceRotationDegrees=0, isOppositeFacing=true, result=0
2021-03-22 14:48:38.775 5668-5855/com.google.mediapipe.apps.objectdetectiongpu D/CameraXPreviewHelper: Received surface request for resolution 1600x1200
2021-03-22 14:48:38.776 5668-5711/com.google.mediapipe.apps.objectdetectiongpu D/UseCaseAttachState: Active and attached use case: [] for camera: 0
2021-03-22 14:48:38.777 5668-5711/com.google.mediapipe.apps.objectdetectiongpu D/UseCaseAttachState: All use case: [androidx.camera.core.Preview-797af778-6a1b-4f90-8f08-e47e3d04d2b840699936] for camera: 0
2021-03-22 14:48:38.778 5668-5711/com.google.mediapipe.apps.objectdetectiongpu D/UseCaseAttachState: Active and attached use case: [androidx.camera.core.Preview-797af778-6a1b-4f90-8f08-e47e3d04d2b840699936] for camera: 0
2021-03-22 14:48:38.781 5668-5855/com.google.mediapipe.apps.objectdetectiongpu D/CameraXPreviewHelper: Providing surface
2021-03-22 14:48:38.782 5668-5711/com.google.mediapipe.apps.objectdetectiongpu D/CameraStateRegistry: tryOpenCamera(Camera@1600186[id=0]) [Available Cameras: 1, Already Open: false (Previous state: CLOSED)] --> SUCCESS
2021-03-22 14:48:38.784 5668-5711/com.google.mediapipe.apps.objectdetectiongpu D/CameraStateRegistry: Recalculating open cameras:
    Camera                                       State                 
    -------------------------------------------------------------------
    Camera@1600186[id=0]                         OPENING               
    Camera@baab012[id=1]                         UNKNOWN               
    -------------------------------------------------------------------
    Open count: 1 (Max allowed: 1)
2021-03-22 14:48:38.785 5668-5711/com.google.mediapipe.apps.objectdetectiongpu D/UseCaseAttachState: All use case: [androidx.camera.core.Preview-797af778-6a1b-4f90-8f08-e47e3d04d2b840699936] for camera: 0
2021-03-22 14:48:38.798 5668-5711/com.google.mediapipe.apps.objectdetectiongpu D/UseCaseAttachState: Active and attached use case: [androidx.camera.core.Preview-797af778-6a1b-4f90-8f08-e47e3d04d2b840699936] for camera: 0
2021-03-22 14:48:38.799 5668-5710/com.google.mediapipe.apps.objectdetectiongpu D/mali_winsys: EGLint new_window_surface(egl_winsys_display *, void *, EGLSurface, EGLConfig, egl_winsys_surface **, EGLBoolean) returns 0x3000
2021-03-22 14:48:38.800 5668-5711/com.google.mediapipe.apps.objectdetectiongpu D/CameraStateRegistry: Recalculating open cameras:
    Camera                                       State                 
    -------------------------------------------------------------------
    Camera@1600186[id=0]                         OPEN                  
    Camera@baab012[id=1]                         UNKNOWN               
    -------------------------------------------------------------------
    Open count: 1 (Max allowed: 1)
2021-03-22 14:48:38.801 5668-5711/com.google.mediapipe.apps.objectdetectiongpu D/UseCaseAttachState: All use case: [androidx.camera.core.Preview-797af778-6a1b-4f90-8f08-e47e3d04d2b840699936] for camera: 0
2021-03-22 14:48:38.802 5668-5711/com.google.mediapipe.apps.objectdetectiongpu D/DeferrableSurface: New surface in use[total_surfaces=1, used_surfaces=1](androidx.camera.core.SurfaceRequest$2@7b2667f}
2021-03-22 14:48:38.802 5668-5711/com.google.mediapipe.apps.objectdetectiongpu D/DeferrableSurface: use count+1, useCount=1 androidx.camera.core.SurfaceRequest$2@7b2667f
2021-03-22 14:48:38.802 5668-5711/com.google.mediapipe.apps.objectdetectiongpu D/CaptureSession: Opening capture session.
2021-03-22 14:48:38.884 5668-5711/com.google.mediapipe.apps.objectdetectiongpu D/CaptureSession: Attempting to send capture request onConfigured
2021-03-22 14:48:38.884 5668-5711/com.google.mediapipe.apps.objectdetectiongpu D/CaptureSession: Issuing request for session.
2021-03-22 14:48:38.885 5668-5711/com.google.mediapipe.apps.objectdetectiongpu D/CaptureSession: CameraCaptureSession.onConfigured() mState=OPENED
2021-03-22 14:48:38.885 5668-5711/com.google.mediapipe.apps.objectdetectiongpu D/CaptureSession: CameraCaptureSession.onReady() OPENED
2021-03-22 14:48:39.215 5668-5855/com.google.mediapipe.apps.objectdetectiongpu D/CameraOrientationUtil: getRelativeImageRotation: destRotationDegrees=0, sourceRotationDegrees=0, isOppositeFacing=true, result=0
2021-03-22 14:48:39.237 5668-5853/com.google.mediapipe.apps.objectdetectiongpu D/mali_winsys: EGLint new_window_surface(egl_winsys_display *, void *, EGLSurface, EGLConfig, egl_winsys_surface **, EGLBoolean) returns 0x3000
2021-03-22 14:48:39.248 5668-5854/com.google.mediapipe.apps.objectdetectiongpu D/ExternalTextureConv: Created output texture: 2 width: 1600 height: 1200
2021-03-22 14:48:39.256 5668-5854/com.google.mediapipe.apps.objectdetectiongpu I/native: I20210322 14:48:39.256675  5854 graph.cc:411] Start running the graph, waiting for inputs.
2021-03-22 14:48:39.256 5668-5854/com.google.mediapipe.apps.objectdetectiongpu I/native: I20210322 14:48:39.256820  5854 gl_context_egl.cc:163] Successfully initialized EGL. Major : 1 Minor: 4
2021-03-22 14:48:39.263 5668-5869/com.google.mediapipe.apps.objectdetectiongpu I/native: I20210322 14:48:39.263001  5869 gl_context.cc:331] GL version: 3.2 (OpenGL ES 3.2 v1.r16p0-01rel0.45b161e1b2cff94b03faf73ffeb6b9a5)
2021-03-22 14:48:39.279 5668-5854/com.google.mediapipe.apps.objectdetectiongpu D/ExternalTextureConv: Created output texture: 3 width: 1600 height: 1200
sgowroji commented 3 years ago

Hi @yangcheng, Could you please share what device/ model are you using to test.
And please share any code changes, if you performed on it with a repo link to understand better. Thanks !

yangcheng commented 3 years ago

@sgowroji It's a custom pad, based on amlogic 311d and android 9 32 bit. Is there some command I could run to give you more information?
Mlkit vision quickstart (https://github.com/googlesamples/mlkit/tree/master/android/vision-quickstart ) can run just fine.

To help isolate the problem I did not do any change and can still reproduce the issue. I build from my fork https://github.com/yangcheng/mediapipe which is even with current master at a92cff7a60031f5c3097b06e74416732d85b5011

I build with 

➜  mediapipe git:(master) ✗ bazel build -c opt --config=android_arm mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu

Starting local Bazel server and connecting to it...
DEBUG: /private/var/tmp/_bazel_chengyang/499bcac60d39d6fdb5899bef0a838c2d/external/rules_foreign_cc/workspace_definitions.bzl:6:6: `@rules_foreign_cc//:workspace_definitions.bzl` has been replaced by `@rules_foreign_cc//foreign_cc:repositories.bzl`. Please use the updated source location
DEBUG: /private/var/tmp/_bazel_chengyang/499bcac60d39d6fdb5899bef0a838c2d/external/rules_foreign_cc/tools/build_defs/deprecation.bzl:5:10: `@rules_foreign_cc//tools/build_defs/...` is deprecated, please find the relevant symbols in `@rules_foreign_cc//foreign_cc/...`. Note that the core rules can now be loaded from `@rules_foreign_cc//foreign_cc:defs.bzl`
INFO: Analyzed target //mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu:objectdetectiongpu (171 packages loaded, 15421 targets configured).
INFO: Found 1 target...
Target //mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu:objectdetectiongpu up-to-date:
  bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu/objectdetectiongpu_deploy.jar
  bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu/objectdetectiongpu_unsigned.apk
  bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu/objectdetectiongpu.apk
INFO: Elapsed time: 25.171s, Critical Path: 5.06s
INFO: 5 processes: 1 internal, 1 darwin-sandbox, 3 worker.
INFO: Build completed successfully, 5 total actions
eknight7 commented 3 years ago

Hi @yangcheng - are you having trouble running other CPU examples as well? For example, does Hand tracking work? Does the basic Hello world Android MediaPipe demo work?

yangcheng commented 3 years ago

I have trouble with other CPU examples as well.

eknight7 commented 3 years ago

Hi - does the basic Hello world Android MediaPipe demo work? This app uses an edge detection graph on GPU. You mentioned

objectdectiongpu can open the camera with a longer delay than mlkit. where should I start to debug the problem? Thanks

so I want to know if the camera lag is happening on simple edge detection graph also.

yangcheng commented 3 years ago

yes, the basic hello world works.

eknight7 commented 3 years ago

Okay if basic hello world works, then we can confirm that the camera preview works for a simple MediaPipe graph. It's possible that the models are heavy for your device configuration - especially since they don't run on CPU but do work slowly on GPU as described in your comments above. Lets confirm if that is indeed the case. Can you try using https://google.github.io/mediapipe/tools/tracing_and_profiling.html#tracing-on-android to collect traces for ObjectDetectionCPU and ObjectDetectionGPU and share the results on your device here?

Also adding @hadon for discussion about using tracing/profiling to determine if the models are too slow for your device.

sgowroji commented 3 years ago

Hi @yangcheng, Could you please provide the traces as mentioned in the above comment. Thanks!

yangcheng commented 3 years ago

I can not reliably reproduce the issue. I'll try more time today and see if I can get anything

sgowroji commented 3 years ago

Hi @yangcheng, Did you get a chance to reproduce the above issue. Thanks!

sgowroji commented 3 years ago

Hi @yangcheng, Could you please respond to my above comment. Thanks!

yangcheng commented 3 years ago

sorry for the late reply, we had a national holiday last week. I reproduced it once. I'll try to get the trace

yangcheng commented 3 years ago

it's unclear to me where to add the profiler config. the doc says

Add the following protobuf message into the existing calculator-graph-config protobuf, such as the existing .pbtxt file. Follow the instructions stated above in Enable tracing

but there is no following message.

hadon commented 3 years ago

I think the doc attempts to refer to the profiler_config message in the section "Enabling tracing and profiling", to avoid repeating that example protobuf message in two sections. It instead should say "add the above ProfilerConfig protobuf message". The config should look like:

profiler_config {
  trace_enabled: true
  enable_profiler: true
  trace_log_interval_count: 200
  trace_log_path: "/sdcard/Download/"
}

Currently, the profiler config can only be specified in the CalculatorGraphConfig for the graph we wish to trace. The field is CalculatorGraphConfig.profiler_config. Let me know if this field is sufficient for you to start profiling and/or tracing.

sgowroji commented 3 years ago

Hi @yangcheng, Please go through the above comment. As the primary query answered we are closing this issue. Feel free to reach us if you're issue still exists or raise another issue for new queries. Thanks!