Object tracking AAR generation and use

Please make sure that this is a solution issue.

System information (Please provide as much relevant information as possible)

Have I written custom code (as opposed to using a stock example script provided in Mediapipe):

OS Platform and Distribution (e.g., Linux Ubuntu 16.04, Android 11, iOS 14.4): Ubuntu 20.04

MediaPipe version: v0.8.8

Bazel version: 3.7.2

Solution (e.g. FaceMesh, Pose, Holistic): Object tracking Android (AAR)

Programming Language and version ( e.g. C++, Python, Java): Java

Describe the expected behavior:

I want to create an object tracking Android application using Mediapipe, and I need the data from the recognition (object id, object type, bounding box...)

Standalone code you may have used to try to get what you need :

Based on the MediaPipe Android Archibe documentation, I created a BUILD file in mediapipe/examples/android/src/java/com/google/mediapipe/apps/object_tracking_aar/BUILD with the content of:

load("//mediapipe/java/com/google/mediapipe:mediapipe_aar.bzl", "mediapipe_aar") mediapipe_aar( name = "mediapipe_object_tracking", calculators = ["//mediapipe/graphs/tracking:mobile_calculators"], )

Than I run this command to generate the aar file:

bazel build -c opt --strip=ALWAYS \ --host_crosstool_top=@bazel_tools//tools/cpp:toolchain \ --fat_apk_cpu=arm64-v8a \ //mediapipe/examples/android/src/java/com/google/mediapipe/apps/object_tracking_aar:mediapipe_object_tracking.aar

Also, I generated the needed .binarypb file by runing:

bazel build -c opt mediapipe/graphs/tracking:mobile_gpu_binary_graph

I tried to modify @jiuqiant MediaPipe face detection aar example by changing the mediapipe_face_detection.aar to the newly generated aar file, and also added the mobile_gpu.binarypb and ssdlite_object_detection.tflite files to the assets.

If I run it on my mobile, it exits after a short black screen.

If there is a problem, provide a reproducible test case that is the bare minimum necessary to generate the problem. If possible, please share a link to Colab/repo link /any notebook: Github

Other info / Complete Logs : Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached:

11/14 21:54:56: Launching 'app' on OnePlus ONEPLUS A5010. App restart successful without requiring a re-install. $ adb shell am start -n "com.example.myfacedetectionapp/com.example.myfacedetectionapp.MainActivity" -a android.intent.action.MAIN -c android.intent.category.LAUNCHER Connected to process 5646 on device 'oneplus-oneplus_a5010-******'. Capturing and displaying logcat messages from application. This behavior can be disabled in the "Logcat output" section of the "Debugger" settings page. I/Perf: Connecting to perf service. I/acedetectionap: [GL_OOM] ClampGrowthLimit 268435456 V/Font: Change font:2 V/Font: Default family:android.graphics.Typeface@64a9e7b9 V/FlingOptimizerScroller: FlingOptimizerOverScroller Init W/acedetectionap: Accessing hidden method Landroid/view/View;->computeFitSystemWindows(Landroid/graphics/Rect;Landroid/graphics/Rect;)Z (greylist, reflection, allowed) W/acedetectionap: Accessing hidden method Landroid/view/ViewGroup;->makeOptionalFitsSystemWindows()V (greylist, reflection, allowed) I/native: I20211114 21:54:56.907318 5646 asset_manager_util.cc:59] Created global reference to asset manager. I/AdrenoGLES: QUALCOMM build : 6209c5d, Ifdcc64d6e8 Build Date : 12/12/19 OpenGL ES Shader Compiler Version: EV031.27.05.03 Local Branch : mybranchea1c9b4a-9e05-f149-a3b2-1458d0b52ae8 Remote Branch : quic/gfx-adreno.lnx.1.0.r85-rel Remote Branch : NONE Reconstruct Branch : NOTHING Build Config : C O 9.0.3 AArch64 I/AdrenoGLES: PFP: 0x005ff112, ME: 0x005ff066 E/libEGL: validate_display:91 error 3008 (EGL_BAD_DISPLAY) I/native: I20211114 21:54:56.932811 5646 gl_context_egl.cc:162] Successfully initialized EGL. Major : 1 Minor: 5 I/native: I20211114 21:54:56.934886 6514 gl_context.cc:334] GL version: 3.2 (OpenGL ES 3.2 V@415.0 (GIT@6209c5d, Ifdcc64d6e8, 1576143393) (Date:12/12/19)) D/PermissionHelper: checkAndRequestCameraPermissions D/GlThread: Starting GL thread ExternalTextureConverter I/CameraManagerGlobal: Connecting to camera service W/CameraManagerGlobal: [soar.cts] ignore the status update of camera: 2 [soar.cts] ignore the status update of camera: 3 W/CameraManagerGlobal: ignore the torch status update of camera: 2 ignore the torch status update of camera: 3 D/CameraRepository: Added camera: 0 D/CameraXPreviewHelper: Camera target size ratio: 1.777778 width: 1280 Camera size candidate width: 4608 height: 3456 ratio: 1.333333 cost: 13648.000000 D/CameraXPreviewHelper: Camera size candidate width: 4608 height: 2592 ratio: 1.777778 cost: 3328.000000 Camera size candidate width: 4608 height: 2304 ratio: 2.000000 cost: 3328.000000 Camera size candidate width: 4160 height: 3120 ratio: 1.333333 cost: 13200.000000 Camera size candidate width: 3840 height: 2160 ratio: 1.777778 cost: 2560.000000 D/CameraXPreviewHelper: Camera size candidate width: 3840 height: 1920 ratio: 2.000000 cost: 2560.000000 Camera size candidate width: 3456 height: 3456 ratio: 1.000000 cost: 12736.000000 Camera size candidate width: 3264 height: 2448 ratio: 1.333333 cost: 12304.000000 Camera size candidate width: 2592 height: 1940 ratio: 1.336082 cost: 11630.020619 D/CameraXPreviewHelper: Camera size candidate width: 2592 height: 1944 ratio: 1.333333 cost: 11632.000000 Camera size candidate width: 2560 height: 1920 ratio: 1.333333 cost: 11600.000000 Camera size candidate width: 2304 height: 1728 ratio: 1.333333 cost: 11344.000000 Camera size candidate width: 2304 height: 1296 ratio: 1.777778 cost: 1024.000000 D/CameraXPreviewHelper: Camera size candidate width: 2304 height: 1152 ratio: 2.000000 cost: 1024.000000 Camera size candidate width: 1840 height: 1380 ratio: 1.333333 cost: 10880.000000 Camera size candidate width: 1920 height: 1080 ratio: 1.777778 cost: 640.000000 Camera size candidate width: 1728 height: 1728 ratio: 1.000000 cost: 11008.000000 D/CameraXPreviewHelper: Camera size candidate width: 1440 height: 1080 ratio: 1.333333 cost: 10480.000000 Camera size candidate width: 1280 height: 960 ratio: 1.333333 cost: 10320.000000 Camera size candidate width: 1280 height: 720 ratio: 1.777778 cost: 0.000000 Camera size candidate width: 1280 height: 480 ratio: 2.666667 cost: 10640.000000 D/CameraXPreviewHelper: Camera size candidate width: 1024 height: 768 ratio: 1.333333 cost: 10576.000000 Camera size candidate width: 720 height: 480 ratio: 1.500000 cost: 10760.000000 Camera size candidate width: 640 height: 480 ratio: 1.333333 cost: 10960.000000 Camera size candidate width: 480 height: 360 ratio: 1.333333 cost: 11120.000000 D/CameraXPreviewHelper: Camera size candidate width: 480 height: 320 ratio: 1.500000 cost: 11000.000000 Camera size candidate width: 352 height: 288 ratio: 1.222222 cost: 11328.000000 Camera size candidate width: 320 height: 240 ratio: 1.333333 cost: 11280.000000 Camera size candidate width: 176 height: 144 ratio: 1.222222 cost: 11504.000000 D/CameraXPreviewHelper: Optimal camera size width: 1280 height: 720 I/Camera2CameraInfo: Device Level: INFO_SUPPORTED_HARDWARE_LEVEL_3 D/CameraRepository: Added camera: 1 D/UseCaseAttachState: Active and attached use case: [] for camera: 0 I/Camera2CameraInfo: Device Level: INFO_SUPPORTED_HARDWARE_LEVEL_3 D/UseCaseAttachState: Active and attached use case: [] for camera: 1 V/ViewRootImpl: The specified message queue synchronization barrier token has not been posted or has already been removed D/DecorView: onWindowFocusChangedFromViewRoot hasFocus: true, DecorView@7ebae7f[MainActivity] D/ViewRootImpl[MainActivity]: windowFocusChanged hasFocus=true inTouchMode=true W/Gralloc3: mapper 3.x is not supported D/CameraOrientationUtil: getRelativeImageRotation: destRotationDegrees=0, sourceRotationDegrees=270, isOppositeFacing=false, result=270 D/DeferrableSurface: Surface created[total_surfaces=1, used_surfaces=0](androidx.camera.core.SurfaceRequest$2@ac8f457} D/CameraXPreviewHelper: Received surface request for resolution 1280x720 D/CameraOrientationUtil: getRelativeImageRotation: destRotationDegrees=0, sourceRotationDegrees=270, isOppositeFacing=false, result=270 D/UseCaseAttachState: Active and attached use case: [] for camera: 1 D/UseCaseAttachState: All use case: [androidx.camera.core.Preview-9697ef14-1e78-411a-87f0-5ff4c820524595640984] for camera: 1 D/CameraXPreviewHelper: Providing surface D/UseCaseAttachState: Active and attached use case: [androidx.camera.core.Preview-9697ef14-1e78-411a-87f0-5ff4c820524595640984] for camera: 1 D/CameraStateRegistry: tryOpenCamera(Camera@5563020[id=1]) [Available Cameras: 1, Already Open: false (Previous state: null)] --> SUCCESS D/CameraStateRegistry: Recalculating open cameras: Camera State ------------------------------------------------------------------- Camera@5563020[id=1] OPENING Camera@46f91c6[id=0] UNKNOWN ------------------------------------------------------------------- Open count: 1 (Max allowed: 1) D/UseCaseAttachState: All use case: [androidx.camera.core.Preview-9697ef14-1e78-411a-87f0-5ff4c820524595640984] for camera: 1 E/libc: Access denied finding property "persist.vendor.camera.privapp.list" W/CameraX-core_ca: type=1400 audit(0.0:179026): avc: denied { read } for name="u:object_r:persist_camera_prop:s0" dev="tmpfs" ino=21583 scontext=u:r:untrusted_app:s0:c191,c257,c512,c768 tcontext=u:object_r:persist_camera_prop:s0 tclass=file permissive=0 D/OnePlusJankManager: Chor uploadMDM JANK_TYPE_ONCE mViewTitle = com.example.myfacedetectionapp/com.example.myfacedetectionapp.MainActivity--- jank level = 1 D/UseCaseAttachState: Active and attached use case: [androidx.camera.core.Preview-9697ef14-1e78-411a-87f0-5ff4c820524595640984] for camera: 1 D/CameraStateRegistry: Recalculating open cameras: Camera State ------------------------------------------------------------------- Camera@5563020[id=1] OPEN Camera@46f91c6[id=0] UNKNOWN ------------------------------------------------------------------- Open count: 1 (Max allowed: 1) D/UseCaseAttachState: All use case: [androidx.camera.core.Preview-9697ef14-1e78-411a-87f0-5ff4c820524595640984] for camera: 1 D/DeferrableSurface: New surface in use[total_surfaces=1, used_surfaces=1](androidx.camera.core.SurfaceRequest$2@ac8f457} use count+1, useCount=1 androidx.camera.core.SurfaceRequest$2@ac8f457 D/CaptureSession: Opening capture session. D/CaptureSession: Attempting to send capture request onConfigured Issuing request for session. D/CaptureSession: CameraCaptureSession.onConfigured() mState=OPENED D/CaptureSession: CameraCaptureSession.onReady() OPENED W/Gralloc3: allocator 3.x is not supported D/ExternalTextureConv: Created output texture: 3 width: 1280 height: 720 I/native: I20211114 21:54:57.474753 6516 graph.cc:476] Start running the graph, waiting for inputs. I20211114 21:54:57.474906 6516 gl_context_egl.cc:162] Successfully initialized EGL. Major : 1 Minor: 5 I/native: I20211114 21:54:57.477075 6591 gl_context.cc:334] GL version: 3.2 (OpenGL ES 3.2 V@415.0 (GIT@6209c5d, Ifdcc64d6e8, 1576143393) (Date:12/12/19)) I/native: I20211114 21:54:57.477245 6516 gl_context_egl.cc:162] Successfully initialized EGL. Major : 1 Minor: 5 I/native: I20211114 21:54:57.478586 6592 gl_context.cc:334] GL version: 3.2 (OpenGL ES 3.2 V@415.0 (GIT@6209c5d, Ifdcc64d6e8, 1576143393) (Date:12/12/19)) W/native: W20211114 21:54:57.481714 6585 motion_analysis_calculator.cc:350] No input video header found. Downstream calculators expecting video headers are likely to fail. D/ExternalTextureConv: Created output texture: 4 width: 1280 height: 720 I/native: I20211114 21:54:57.548188 6514 resource_util_android.cc:89] Successfully loaded: ssdlite_object_detection.tflite I/tflite: Initialized TensorFlow Lite runtime. Created TensorFlow Lite delegate for GPU. I/tflite: Replacing 114 node(s) with delegate (TfLiteGpuDelegate) node, yielding 1 partitions. D/ExternalTextureConv: Created output texture: 6 width: 1280 height: 720 D/ExternalTextureConv: Created output texture: 7 width: 1280 height: 720 D/ExternalTextureConv: Created output texture: 8 width: 1280 height: 720 D/ExternalTextureConv: Created output texture: 9 width: 1280 height: 720 D/ExternalTextureConv: Created output texture: 10 width: 1280 height: 720 D/ExternalTextureConv: Created output texture: 11 width: 1280 height: 720 D/ExternalTextureConv: Created output texture: 12 width: 1280 height: 720 D/ExternalTextureConv: Created output texture: 13 width: 1280 height: 720 D/ExternalTextureConv: Created output texture: 14 width: 1280 height: 720 D/ExternalTextureConv: Created output texture: 15 width: 1280 height: 720 D/ExternalTextureConv: Created output texture: 24 width: 1280 height: 720 D/ExternalTextureConv: Created output texture: 41 width: 1280 height: 720 D/ExternalTextureConv: Created output texture: 62 width: 1280 height: 720 D/ExternalTextureConv: Created output texture: 89 width: 1280 height: 720 D/ExternalTextureConv: Created output texture: 112 width: 1280 height: 720 D/ExternalTextureConv: Created output texture: 133 width: 1280 height: 720 D/ExternalTextureConv: Created output texture: 146 width: 1280 height: 720 D/ExternalTextureConv: Created output texture: 159 width: 1280 height: 720 D/ExternalTextureConv: Created output texture: 170 width: 1280 height: 720 D/ExternalTextureConv: Created output texture: 179 width: 1280 height: 720 D/ExternalTextureConv: Created output texture: 192 width: 1280 height: 720 D/ExternalTextureConv: Created output texture: 203 width: 1280 height: 720 A/libc: Fatal signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0x0 in tid 6587 (mediapipe/6587), pid 5646 (acedetectionapp) W/native: W20211114 21:54:58.895328 6590 motion_analysis_calculator.cc:743] Requested image format in RegionFlowComputation does not match video stream format. Overriding. W/native: W20211114 21:54:58.898322 6590 parallel_invoker.h:241] Unsupported invoker mode selected on Android. Falling back to ThreadPool D/ExternalTextureConv: Created output texture: 252 width: 1280 height: 720 D/ExternalTextureConv: Created output texture: 253 width: 1280 height: 720 D/ExternalTextureConv: Created output texture: 254 width: 1280 height: 720 D/ExternalTextureConv: Created output texture: 255 width: 1280 height: 720 D/ExternalTextureConv: Created output texture: 256 width: 1280 height: 720 D/ExternalTextureConv: Created output texture: 257 width: 1280 height: 720 D/ExternalTextureConv: Created output texture: 258 width: 1280 height: 720

google-ai-edge / mediapipe

Object tracking AAR generation and use #2769