search

google-ai-edge / mediapipe

Cross-platform, customizable ML solutions for live and streaming media.
https://mediapipe.dev
Apache License 2.0
26.78k stars 5.09k forks source link

how to use nnapi in the face_landmark_cpu.pbtxt #4127

Open taeyeonlee opened 1 year ago

taeyeonlee commented 1 year ago

When building the mediapipe Holistic AAR, I modified the face_landmark_cpu.pbtxt, like below, to use nnapi.

  calculator: "InferenceCalculator"                     
  input_stream: "TENSORS:input_tensors"                     
  output_stream: "TENSORS:detection_tensors"                        
  options: {                        
    [mediapipe.InferenceCalculatorOptions.ext] {                        
      model_path: "mediapipe/modules/pose_detection/pose_detection.tflite"                      
      delegate {                        
        nnapi {}                            
      }                     
    }                       
  }                     
}

The Android app runs following error. How to fix it ?

I/native: I20230226 21:25:48.154594 10604 resource_util_android.cc:89] Successfully loaded: pose_detection.tflite
D/ExternalTextureConv: Created output texture: 6 width: 1920 height: 1080
I/native: I20230226 21:25:48.156738 10605 resource_util_android.cc:89] Successfully loaded: face_landmark_with_attention.tflite
I/native: I20230226 21:25:48.169183 10602 resource_util_android.cc:89] Successfully loaded: pose_landmark_lite.tflite
I/native: I20230226 21:25:48.173480 10626 jni_util.cc:41] GetEnv: not attached
I/tflite: Initialized TensorFlow Lite runtime.
I/native: I20230226 21:25:48.175982 10600 jni_util.cc:41] GetEnv: not attached
I/native: I20230226 21:25:48.177188 10606 resource_util_android.cc:89] Successfully loaded: hand_landmark_full.tflite
I/native: I20230226 21:25:48.187690 10603 resource_util_android.cc:89] Successfully loaded: hand_landmark_full.tflite
D/ExternalTextureConv: Created output texture: 7 width: 1920 height: 1080
I/native: I20230226 21:25:48.206269 10606 jni_util.cc:41] GetEnv: not attached
I/native: I20230226 21:25:48.252709 10599 jni_util.cc:41] GetEnv: not attached
I/tflite: Created TensorFlow Lite delegate for NNAPI.
I/Manager: DeviceManager::DeviceManager
I/Manager: findAvailableDevices
I/Manager: Found interface qti-default
I/native: I20230226 21:25:48.258436 10603 jni_util.cc:41] GetEnv: not attached
I/VersionedInterfaces: Capab {.relaxedFloat32toFloat16PerformanceScalar = {.execTime = 0.600000, .powerUsage = 0.800000}, .relaxedFloat32toFloat16PerformanceTensor = {.execTime = 0.600000, .powerUsage = 0.800000}, .operandPerformance = [15]{{.type = FLOAT32, .info = {.execTime = 0.600000, .powerUsage = 0.800000}}, {.type = INT32, .info = {.execTime = 0.600000, .powerUsage = 0.800000}}, {.type = UINT32, .info = {.execTime = 0.600000, .powerUsage = 0.800000}}, {.type = TENSOR_FLOAT32, .info = {.execTime = 0.600000, .powerUsage = 0.800000}}, {.type = TENSOR_INT32, .info = {.execTime = 0.600000, .powerUsage = 0.800000}}, {.type = TENSOR_QUANT8_ASYMM, .info = {.execTime = 0.700000, .powerUsage = 0.700000}}, {.type = BOOL, .info = {.execTime = 0.600000, .powerUsage = 0.800000}}, {.type = TENSOR_QUANT16_SYMM, .info = {.execTime = 0.700000, .powerUsage = 0.700000}}, {.type = TENSOR_FLOAT16, .info = {.execTime = 0.600000, .powerUsage = 0.800000}}, {.type = TENSOR_BOOL8, .info = {.execTime = 0.600000, .powerUsage = 0.800000}}, {.type = FLOAT16, .info = {.execTime = 0.600000, .powerUsage = 0.800000}}, {.type = TENSOR_QUANT8_SYMM_PER_CHANNEL, .info = {.execTime = 0.700000, .powerUsage = 0.700000}}, {.type = TENSOR_QUANT16_ASYMM, .info = {.execTime = 0.700000, .powerUsage = 0.700000}}, {.type = TENSOR_QUANT8_SYMM, .info = {.execTime = 0.700000, .powerUsage = 0.700000}}, {.type = TENSOR_QUANT8_ASYMM_SIGNED, .info = {.execTime = 0.700000, .powerUsage = 0.700000}}}, .ifPerformance = {.execTime = 0.600000, .powerUsage = 0.800000}, .whilePerformance = {.execTime = 0.600000, .powerUsage = 0.800000}}
I/Manager: Found interface qti-dsp
I/VersionedInterfaces: Capab {.relaxedFloat32toFloat16PerformanceScalar = {.execTime = 2.000000, .powerUsage = 2.000000}, .relaxedFloat32toFloat16PerformanceTensor = {.execTime = 2.000000, .powerUsage = 2.000000}, .operandPerformance = [15]{{.type = FLOAT32, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = INT32, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = UINT32, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_FLOAT32, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_INT32, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_QUANT8_ASYMM, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = BOOL, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_QUANT16_SYMM, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_FLOAT16, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_BOOL8, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = FLOAT16, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_QUANT8_SYMM_PER_CHANNEL, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_QUANT16_ASYMM, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_QUANT8_SYMM, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_QUANT8_ASYMM_SIGNED, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}}, .ifPerformance = {.execTime = 2.000000, .powerUsage = 2.000000}, .whilePerformance = {.execTime = 2.000000, .powerUsage = 2.000000}}
I/Manager: Found interface qti-gpu
I/VersionedInterfaces: Capab {.relaxedFloat32toFloat16PerformanceScalar = {.execTime = 1.100000, .powerUsage = 1.100000}, .relaxedFloat32toFloat16PerformanceTensor = {.execTime = 1.100000, .powerUsage = 1.100000}, .operandPerformance = [15]{{.type = FLOAT32, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = INT32, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = UINT32, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_FLOAT32, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_INT32, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_QUANT8_ASYMM, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = BOOL, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_QUANT16_SYMM, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_FLOAT16, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_BOOL8, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = FLOAT16, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_QUANT8_SYMM_PER_CHANNEL, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_QUANT16_ASYMM, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_QUANT8_SYMM, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_QUANT8_ASYMM_SIGNED, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}}, .ifPerformance = {.execTime = 1.100000, .powerUsage = 1.100000}, .whilePerformance = {.execTime = 1.100000, .powerUsage = 1.100000}}
I/Manager: Found interface qti-hta
I/VersionedInterfaces: Capab {.relaxedFloat32toFloat16PerformanceScalar = {.execTime = 2.000000, .powerUsage = 2.000000}, .relaxedFloat32toFloat16PerformanceTensor = {.execTime = 2.000000, .powerUsage = 2.000000}, .operandPerformance = [15]{{.type = FLOAT32, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = INT32, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = UINT32, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_FLOAT32, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_INT32, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_QUANT8_ASYMM, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = BOOL, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_QUANT16_SYMM, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_FLOAT16, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_BOOL8, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = FLOAT16, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_QUANT8_SYMM_PER_CHANNEL, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_QUANT16_ASYMM, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_QUANT8_SYMM, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_QUANT8_ASYMM_SIGNED, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}}, .ifPerformance = {.execTime = 2.000000, .powerUsage = 2.000000}, .whilePerformance = {.execTime = 2.000000, .powerUsage = 2.000000}}
I/TypeManager: Failed to read /vendor/etc/nnapi_extensions_app_allowlist ; No app allowlisted for vendor extensions use.
I/native: I20230226 21:25:48.322476 10601 jni_util.cc:41] GetEnv: not attached
D/ExternalTextureConv: Created output texture: 8 width: 1920 height: 1080
I/native: I20230226 21:25:48.435539 10605 jni_util.cc:41] GetEnv: not attached
I/native: I20230226 21:25:48.478067 10602 jni_util.cc:41] GetEnv: not attached
I/native: I20230226 21:25:48.762856 10602 resource_util_android.cc:89] Successfully loaded: face_detection_short_range.tflite
I/native: I20230226 21:25:48.764724 10605 resource_util_android.cc:89] Successfully loaded: hand_recrop.tflite
I/native: I20230226 21:25:48.764724 10599 resource_util_android.cc:89] Successfully loaded: hand_recrop.tflite
I/tflite: Created TensorFlow Lite XNNPACK delegate for CPU.
I/tflite: Replacing 162 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 2 partitions.
E/OperationsUtils: NN_RET_CHECK failed (frameworks/ml/nn/common/OperationsUtils.cpp:43): type == expectedTypes[i] Invalid input tensor type TENSOR_FLOAT16 for input 1, expected TENSOR_FLOAT32
E/Utils: Validation failed for operation CONV_2D
E/tflite: NN API returned error ANEURALNETWORKS_BAD_DATA at line 1137 while adding operation.
E/tflite: Restored original execution plan after delegate application failure.
I/tflite: Replacing 63 node(s) with delegate (TfLiteNnapiDelegate) node, yielding 1 partitions.
E/FrameProcessor: Mediapipe error: 
    com.google.mediapipe.framework.MediaPipeException: internal: Graph has errors: 
    Calculator::Open() for node "holisticlandmarkcpu__facelandmarksfromposecpu__facelandmarkcpu__inferencecalculator__holisticlandmarkcpu__facelandmarksfromposecpu__facelandmarkcpu__InferenceCalculator" failed: ; (interpreter_->ModifyGraphWithDelegate(delegate_.get()))==(kTfLiteOk)cpu.cc:217) 
        at com.google.mediapipe.framework.Graph.nativeMovePacketToInputStream(Native Method)
        at com.google.mediapipe.framework.Graph.addConsumablePacketToInputStream(Graph.java:395)
        at com.google.mediapipe.components.FrameProcessor.onNewFrame(FrameProcessor.java:458)
        at com.google.mediapipe.components.ExternalTextureConverter$RenderThread.renderNext(ExternalTextureConverter.java:403)
        at com.google.mediapipe.components.ExternalTextureConverter$RenderThread.lambda$onFrameAvailable$0$ExternalTextureConverter$RenderThread(ExternalTextureConverter.java:338)
        at com.google.mediapipe.components.-$$Lambda$ExternalTextureConverter$RenderThread$IXJjtE68JMV2CMJNQcFYhTUqhj8.run(Unknown Source:4)
        at android.os.Handler.handleCallback(Handler.java:938)
        at android.os.Handler.dispatchMessage(Handler.java:99)
        at android.os.Looper.loop(Looper.java:223)
        at com.google.mediapipe.glutil.GlThread.run(GlThread.java:141)
I/tflite: Replacing 63 node(s) with delegate (TfLiteNnapiDelegate) node, yielding 1 partitions.

When using the xnnpack in the face_landmark_cpu.pbtxt, like below. The Android App is working well with the built AAR.


  calculator: "InferenceCalculator"                     
  input_stream: "TENSORS:input_tensors"                     
  output_stream: "TENSORS:detection_tensors"                        
  options: {                        
    [mediapipe.InferenceCalculatorOptions.ext] {                        
      model_path: "mediapipe/modules/pose_detection/pose_detection.tflite"                      
      delegate {                        
        xnnpack {}                          
      }                     
    }                       
  }                     
}

Whole Error log


02/26 21:25:41: Launching 'app'.
Install successfully finished in 5 s 562 ms.
$ adb shell am start -n "com.lite.holistic_tracking/com.lite.holistic_tracking.MainActivity" -a android.intent.action.MAIN -c android.intent.category.LAUNCHER
Connected to process 10471 on device.
Capturing and displaying logcat messages from application. This behavior can be disabled in the "Logcat output" section of the "Debugger" settings page.
I/Typeface: Current DefaultTypeface ( default_lgsmartui )
D/NetworkSecurityConfig: No Network Security Config specified, using platform default
D/NetworkSecurityConfig: No Network Security Config specified, using platform default
W/listic_trackin: Accessing hidden method Landroid/view/View;->computeFitSystemWindows(Landroid/graphics/Rect;Landroid/graphics/Rect;)Z (greylist, reflection, allowed)
W/listic_trackin: Accessing hidden method Landroid/view/ViewGroup;->makeOptionalFitsSystemWindows()V (greylist, reflection, allowed)
D/PermissionHelper: checkAndRequestCameraPermissions
V/holistic_Service: [taeyeon] onCreate 
I/native: I20230226 21:25:47.516883 10471 asset_manager_util.cc:59] Created global reference to asset manager.
I/AdrenoGLES-0: QUALCOMM build                   : e5dfe77, I46bb81ccf7
    Build Date                       : 08/16/20
    OpenGL ES Shader Compiler Version: EV031.32.02.00
    Local Branch                     : 
    Remote Branch                    : 
    Remote Branch                    : 
    Reconstruct Branch               : 
I/AdrenoGLES-0: Build Config                     : S P 10.0.5 AArch64
I/AdrenoGLES-0: Driver Path                      : /vendor/lib64/egl/libGLESv2_adreno.so
I/AdrenoGLES-0: PFP: 0x016dd091, ME: 0x00000000
E/libEGL: validate_display:87 error 3008 (EGL_BAD_DISPLAY)
I/native: I20230226 21:25:47.536614 10471 gl_context_egl.cc:162] Successfully initialized EGL. Major : 1 Minor: 5
I/native: I20230226 21:25:47.537825 10548 gl_context.cc:334] GL version: 3.2 (OpenGL ES 3.2 V@0502.0 (GIT@e5dfe77, I46bb81ccf7, 1597647511) (Date:08/16/20))
D/GlThread: Starting GL thread ExternalTextureConverter
I/holistic_Service: [taeyeon] startCamera()
I/CameraManagerGlobal: Connecting to camera service
I/CameraManagerGlobal: boundary_number_of_camera = 3
I/chatty: uid=10342(com.lite.holistic_tracking) identical 2 lines
I/CameraManagerGlobal: boundary_number_of_camera = 3
W/CameraManagerGlobal: [soar.cts] ignore the status update of camera: 3
I/CameraManagerGlobal: boundary_number_of_camera = 3
W/CameraManagerGlobal: [soar.cts] ignore the status update of camera: 4
I/CameraManagerGlobal: boundary_number_of_camera = 3
W/CameraManagerGlobal: [soar.cts] ignore the status update of camera: 5
I/CameraManagerGlobal: boundary_number_of_camera = 3
W/CameraManagerGlobal: [soar.cts] ignore the status update of camera: 6
I/CameraManagerGlobal: boundary_number_of_camera = 3
W/CameraManagerGlobal: [soar.cts] ignore the status update of camera: 7
I/CameraManagerGlobal: boundary_number_of_camera = 3
W/CameraManagerGlobal: [soar.cts] ignore the status update of camera: 8
I/CameraManagerGlobal: boundary_number_of_camera = 3
W/CameraManagerGlobal: [soar.cts] ignore the status update of camera: 9
I/CameraManagerGlobal: boundary_number_of_camera = 3
I/CameraManager: camera Id = 0, packageName = com.lite.holistic_tracking
I/CameraManagerGlobal: boundary_number_of_camera = 3
I/CameraManagerGlobal: boundary_number_of_camera = 3
W/CameraManagerGlobal: ignore the torch status update of camera: 5
I/CameraManagerGlobal: boundary_number_of_camera = 3
I/CameraManager: camera Id = 0, packageName = com.lite.holistic_tracking
I/CameraManager: camera Id = 1, packageName = com.lite.holistic_tracking
D/CameraXPreviewHelper: Camera target size ratio: 0.562500 width: 1080
I/CameraManager: camera Id = 2, packageName = com.lite.holistic_tracking
D/CameraXPreviewHelper: Camera size candidate width: 4000 height: 3000 ratio: 1.333333 cost: 14400.000000
D/CameraXPreviewHelper: Camera size candidate width: 4000 height: 2250 ratio: 1.777778 cost: 15253.333333
D/CameraXPreviewHelper: Camera size candidate width: 4000 height: 1760 ratio: 2.272727 cost: 16203.636364
D/CameraXPreviewHelper: Camera size candidate width: 3840 height: 2160 ratio: 1.777778 cost: 15093.333333
D/CameraXPreviewHelper: Camera size candidate width: 3264 height: 2448 ratio: 1.333333 cost: 13664.000000
D/CameraXPreviewHelper: Camera size candidate width: 3264 height: 1836 ratio: 1.777778 cost: 14517.333333
I/CameraManagerGlobal: boundary_number_of_camera = 3
D/CameraXPreviewHelper: Camera size candidate width: 3264 height: 1434 ratio: 2.276151 cost: 15474.209205
I/CameraManagerGlobal: boundary_number_of_camera = 3
D/CameraRepository: Added camera: 0
D/CameraXPreviewHelper: Camera size candidate width: 2560 height: 1920 ratio: 1.333333 cost: 12960.000000
I/CameraManagerGlobal: boundary_number_of_camera = 3
D/CameraXPreviewHelper: Camera size candidate width: 2048 height: 1536 ratio: 1.333333 cost: 12448.000000
D/CameraXPreviewHelper: Camera size candidate width: 1920 height: 1080 ratio: 1.777778 cost: 13173.333333
D/CameraXPreviewHelper: Camera size candidate width: 1640 height: 720 ratio: 2.277778 cost: 13853.333333
D/CameraXPreviewHelper: Camera size candidate width: 1440 height: 1080 ratio: 1.333333 cost: 11840.000000
D/CameraXPreviewHelper: Camera size candidate width: 1440 height: 960 ratio: 1.500000 cost: 12160.000000
D/CameraXPreviewHelper: Camera size candidate width: 1408 height: 1152 ratio: 1.222222 cost: 11594.666667
D/CameraXPreviewHelper: Camera size candidate width: 1280 height: 960 ratio: 1.333333 cost: 11680.000000
D/CameraXPreviewHelper: Camera size candidate width: 1280 height: 768 ratio: 1.666667 cost: 12320.000000
D/CameraXPreviewHelper: Camera size candidate width: 1280 height: 720 ratio: 1.777778 cost: 12533.333333
D/CameraXPreviewHelper: Camera size candidate width: 960 height: 720 ratio: 1.333333 cost: 11600.000000
D/CameraXPreviewHelper: Camera size candidate width: 960 height: 540 ratio: 1.777778 cost: 12453.333333
D/CameraXPreviewHelper: Camera size candidate width: 720 height: 720 ratio: 1.000000 cost: 11200.000000
D/CameraXPreviewHelper: Camera size candidate width: 720 height: 540 ratio: 1.333333 cost: 11840.000000
D/CameraXPreviewHelper: Camera size candidate width: 720 height: 480 ratio: 1.500000 cost: 12160.000000
D/CameraXPreviewHelper: Camera size candidate width: 640 height: 480 ratio: 1.333333 cost: 11920.000000
D/CameraXPreviewHelper: Camera size candidate width: 480 height: 640 ratio: 0.750000 cost: 600.000000
D/CameraXPreviewHelper: Camera size candidate width: 352 height: 288 ratio: 1.222222 cost: 11994.666667
D/CameraXPreviewHelper: Camera size candidate width: 320 height: 240 ratio: 1.333333 cost: 12240.000000
D/CameraXPreviewHelper: Camera size candidate width: 176 height: 144 ratio: 1.222222 cost: 12170.666667
D/CameraXPreviewHelper: Optimal camera size width: 480 height: 640
I/CameraManager: camera Id = 0, packageName = com.lite.holistic_tracking
I/Camera2CameraInfo: Device Level: INFO_SUPPORTED_HARDWARE_LEVEL_3
D/CameraRepository: Added camera: 1
I/CameraManagerGlobal: boundary_number_of_camera = 3
I/CameraManager: camera Id = 1, packageName = com.lite.holistic_tracking
I/Camera2CameraInfo: Device Level: INFO_SUPPORTED_HARDWARE_LEVEL_3
D/CameraRepository: Added camera: 2
I/CameraManagerGlobal: boundary_number_of_camera = 3
I/CameraManager: camera Id = 2, packageName = com.lite.holistic_tracking
I/Camera2CameraInfo: Device Level: INFO_SUPPORTED_HARDWARE_LEVEL_3
D/UseCaseAttachState: Active and attached use case: [] for camera: 0
D/UseCaseAttachState: Active and attached use case: [] for camera: 2
D/UseCaseAttachState: Active and attached use case: [] for camera: 1
D/CameraOrientationUtil: getRelativeImageRotation: destRotationDegrees=0, sourceRotationDegrees=90, isOppositeFacing=true, result=90
D/DeferrableSurface: Surface created[total_surfaces=1, used_surfaces=0](androidx.camera.core.SurfaceRequest$2@ce9d28}
D/CameraOrientationUtil: getRelativeImageRotation: destRotationDegrees=0, sourceRotationDegrees=90, isOppositeFacing=true, result=90
D/UseCaseAttachState: Active and attached use case: [] for camera: 0
D/CameraXPreviewHelper: Received surface request for resolution 480x640
D/UseCaseAttachState: All use case: [androidx.camera.core.Preview-cc396a9f-bcb0-4151-ba49-7f232dd6fd41162268869] for camera: 0
D/UseCaseAttachState: Active and attached use case: [androidx.camera.core.Preview-cc396a9f-bcb0-4151-ba49-7f232dd6fd41162268869] for camera: 0
D/CameraXPreviewHelper: Providing surface
D/CameraStateRegistry: tryOpenCamera(Camera@aa07c29[id=0]) [Available Cameras: 1, Already Open: false (Previous state: null)] --> SUCCESS
D/CameraStateRegistry: Recalculating open cameras:
    Camera                                       State                 
    -------------------------------------------------------------------
    Camera@19cb4ba[id=1]                         UNKNOWN               
    Camera@7d16286[id=2]                         UNKNOWN               
    Camera@aa07c29[id=0]                         OPENING               
    -------------------------------------------------------------------
    Open count: 1 (Max allowed: 1)
D/UseCaseAttachState: All use case: [androidx.camera.core.Preview-cc396a9f-bcb0-4151-ba49-7f232dd6fd41162268869] for camera: 0
I/CameraManager: camera Id = 0, packageName = com.lite.holistic_tracking
D/GlThread: Starting GL thread ExternalTextureConverter
I/CameraManagerGlobal: boundary_number_of_camera = 3
E/CameraManagerGlobal: Camera 6 is not available. Ignore physical camera status change
E/CameraManagerGlobal: Camera 7 is not available. Ignore physical camera status change
E/CameraManagerGlobal: Camera 8 is not available. Ignore physical camera status change
E/CameraManagerGlobal: Camera 9 is not available. Ignore physical camera status change
I/CameraManagerGlobal: boundary_number_of_camera = 3
I/CameraManagerGlobal: boundary_number_of_camera = 3
W/CameraManagerGlobal: ignore the torch status update of camera: 5
D/UseCaseAttachState: Active and attached use case: [androidx.camera.core.Preview-cc396a9f-bcb0-4151-ba49-7f232dd6fd41162268869] for camera: 0
D/CameraStateRegistry: Recalculating open cameras:
    Camera                                       State                 
    -------------------------------------------------------------------
    Camera@19cb4ba[id=1]                         UNKNOWN               
    Camera@7d16286[id=2]                         UNKNOWN               
    Camera@aa07c29[id=0]                         OPEN                  
    -------------------------------------------------------------------
    Open count: 1 (Max allowed: 1)
D/UseCaseAttachState: All use case: [androidx.camera.core.Preview-cc396a9f-bcb0-4151-ba49-7f232dd6fd41162268869] for camera: 0
D/DeferrableSurface: New surface in use[total_surfaces=1, used_surfaces=1](androidx.camera.core.SurfaceRequest$2@ce9d28}
D/DeferrableSurface: use count+1, useCount=1 androidx.camera.core.SurfaceRequest$2@ce9d28
D/CaptureSession: Opening capture session.
D/CaptureSession: Attempting to send capture request onConfigured
D/CaptureSession: Issuing request for session.
D/CaptureSession: CameraCaptureSession.onConfigured() mState=OPENED
D/CaptureSession: CameraCaptureSession.onReady() OPENED
D/ExternalTextureConv: Created output texture: 4 width: 1920 height: 1080
I/native: I20230226 21:25:48.124748 10562 graph.cc:476] Start running the graph, waiting for inputs.
I/native: I20230226 21:25:48.124967 10562 gl_context_egl.cc:162] Successfully initialized EGL. Major : 1 Minor: 5
I/native: I20230226 21:25:48.126107 10626 gl_context.cc:334] GL version: 3.2 (OpenGL ES 3.2 V@0502.0 (GIT@e5dfe77, I46bb81ccf7, 1597647511) (Date:08/16/20))
I/native: I20230226 21:25:48.126356 10562 gl_context_egl.cc:162] Successfully initialized EGL. Major : 1 Minor: 5
I/native: I20230226 21:25:48.127363 10627 gl_context.cc:334] GL version: 3.2 (OpenGL ES 3.2 V@0502.0 (GIT@e5dfe77, I46bb81ccf7, 1597647511) (Date:08/16/20))
I/native: I20230226 21:25:48.127540 10562 gl_context_egl.cc:162] Successfully initialized EGL. Major : 1 Minor: 5
I/native: I20230226 21:25:48.128559 10628 gl_context.cc:334] GL version: 3.2 (OpenGL ES 3.2 V@0502.0 (GIT@e5dfe77, I46bb81ccf7, 1597647511) (Date:08/16/20))
D/ExternalTextureConv: Created output texture: 5 width: 1920 height: 1080
I/native: I20230226 21:25:48.154594 10604 resource_util_android.cc:89] Successfully loaded: pose_detection.tflite
D/ExternalTextureConv: Created output texture: 6 width: 1920 height: 1080
I/native: I20230226 21:25:48.156738 10605 resource_util_android.cc:89] Successfully loaded: face_landmark_with_attention.tflite
I/native: I20230226 21:25:48.169183 10602 resource_util_android.cc:89] Successfully loaded: pose_landmark_lite.tflite
I/native: I20230226 21:25:48.173480 10626 jni_util.cc:41] GetEnv: not attached
I/tflite: Initialized TensorFlow Lite runtime.
I/native: I20230226 21:25:48.175982 10600 jni_util.cc:41] GetEnv: not attached
I/native: I20230226 21:25:48.177188 10606 resource_util_android.cc:89] Successfully loaded: hand_landmark_full.tflite
I/native: I20230226 21:25:48.187690 10603 resource_util_android.cc:89] Successfully loaded: hand_landmark_full.tflite
D/ExternalTextureConv: Created output texture: 7 width: 1920 height: 1080
I/native: I20230226 21:25:48.206269 10606 jni_util.cc:41] GetEnv: not attached
I/native: I20230226 21:25:48.252709 10599 jni_util.cc:41] GetEnv: not attached
I/tflite: Created TensorFlow Lite delegate for NNAPI.
I/Manager: DeviceManager::DeviceManager
I/Manager: findAvailableDevices
I/Manager: Found interface qti-default
I/native: I20230226 21:25:48.258436 10603 jni_util.cc:41] GetEnv: not attached
I/VersionedInterfaces: Capab {.relaxedFloat32toFloat16PerformanceScalar = {.execTime = 0.600000, .powerUsage = 0.800000}, .relaxedFloat32toFloat16PerformanceTensor = {.execTime = 0.600000, .powerUsage = 0.800000}, .operandPerformance = [15]{{.type = FLOAT32, .info = {.execTime = 0.600000, .powerUsage = 0.800000}}, {.type = INT32, .info = {.execTime = 0.600000, .powerUsage = 0.800000}}, {.type = UINT32, .info = {.execTime = 0.600000, .powerUsage = 0.800000}}, {.type = TENSOR_FLOAT32, .info = {.execTime = 0.600000, .powerUsage = 0.800000}}, {.type = TENSOR_INT32, .info = {.execTime = 0.600000, .powerUsage = 0.800000}}, {.type = TENSOR_QUANT8_ASYMM, .info = {.execTime = 0.700000, .powerUsage = 0.700000}}, {.type = BOOL, .info = {.execTime = 0.600000, .powerUsage = 0.800000}}, {.type = TENSOR_QUANT16_SYMM, .info = {.execTime = 0.700000, .powerUsage = 0.700000}}, {.type = TENSOR_FLOAT16, .info = {.execTime = 0.600000, .powerUsage = 0.800000}}, {.type = TENSOR_BOOL8, .info = {.execTime = 0.600000, .powerUsage = 0.800000}}, {.type = FLOAT16, .info = {.execTime = 0.600000, .powerUsage = 0.800000}}, {.type = TENSOR_QUANT8_SYMM_PER_CHANNEL, .info = {.execTime = 0.700000, .powerUsage = 0.700000}}, {.type = TENSOR_QUANT16_ASYMM, .info = {.execTime = 0.700000, .powerUsage = 0.700000}}, {.type = TENSOR_QUANT8_SYMM, .info = {.execTime = 0.700000, .powerUsage = 0.700000}}, {.type = TENSOR_QUANT8_ASYMM_SIGNED, .info = {.execTime = 0.700000, .powerUsage = 0.700000}}}, .ifPerformance = {.execTime = 0.600000, .powerUsage = 0.800000}, .whilePerformance = {.execTime = 0.600000, .powerUsage = 0.800000}}
I/Manager: Found interface qti-dsp
I/VersionedInterfaces: Capab {.relaxedFloat32toFloat16PerformanceScalar = {.execTime = 2.000000, .powerUsage = 2.000000}, .relaxedFloat32toFloat16PerformanceTensor = {.execTime = 2.000000, .powerUsage = 2.000000}, .operandPerformance = [15]{{.type = FLOAT32, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = INT32, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = UINT32, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_FLOAT32, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_INT32, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_QUANT8_ASYMM, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = BOOL, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_QUANT16_SYMM, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_FLOAT16, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_BOOL8, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = FLOAT16, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_QUANT8_SYMM_PER_CHANNEL, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_QUANT16_ASYMM, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_QUANT8_SYMM, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_QUANT8_ASYMM_SIGNED, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}}, .ifPerformance = {.execTime = 2.000000, .powerUsage = 2.000000}, .whilePerformance = {.execTime = 2.000000, .powerUsage = 2.000000}}
I/Manager: Found interface qti-gpu
I/VersionedInterfaces: Capab {.relaxedFloat32toFloat16PerformanceScalar = {.execTime = 1.100000, .powerUsage = 1.100000}, .relaxedFloat32toFloat16PerformanceTensor = {.execTime = 1.100000, .powerUsage = 1.100000}, .operandPerformance = [15]{{.type = FLOAT32, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = INT32, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = UINT32, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_FLOAT32, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_INT32, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_QUANT8_ASYMM, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = BOOL, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_QUANT16_SYMM, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_FLOAT16, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_BOOL8, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = FLOAT16, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_QUANT8_SYMM_PER_CHANNEL, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_QUANT16_ASYMM, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_QUANT8_SYMM, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_QUANT8_ASYMM_SIGNED, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}}, .ifPerformance = {.execTime = 1.100000, .powerUsage = 1.100000}, .whilePerformance = {.execTime = 1.100000, .powerUsage = 1.100000}}
I/Manager: Found interface qti-hta
I/VersionedInterfaces: Capab {.relaxedFloat32toFloat16PerformanceScalar = {.execTime = 2.000000, .powerUsage = 2.000000}, .relaxedFloat32toFloat16PerformanceTensor = {.execTime = 2.000000, .powerUsage = 2.000000}, .operandPerformance = [15]{{.type = FLOAT32, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = INT32, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = UINT32, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_FLOAT32, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_INT32, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_QUANT8_ASYMM, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = BOOL, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_QUANT16_SYMM, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_FLOAT16, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_BOOL8, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = FLOAT16, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_QUANT8_SYMM_PER_CHANNEL, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_QUANT16_ASYMM, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_QUANT8_SYMM, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_QUANT8_ASYMM_SIGNED, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}}, .ifPerformance = {.execTime = 2.000000, .powerUsage = 2.000000}, .whilePerformance = {.execTime = 2.000000, .powerUsage = 2.000000}}
I/TypeManager: Failed to read /vendor/etc/nnapi_extensions_app_allowlist ; No app allowlisted for vendor extensions use.
I/native: I20230226 21:25:48.322476 10601 jni_util.cc:41] GetEnv: not attached
D/ExternalTextureConv: Created output texture: 8 width: 1920 height: 1080
I/native: I20230226 21:25:48.435539 10605 jni_util.cc:41] GetEnv: not attached
I/native: I20230226 21:25:48.478067 10602 jni_util.cc:41] GetEnv: not attached
I/native: I20230226 21:25:48.762856 10602 resource_util_android.cc:89] Successfully loaded: face_detection_short_range.tflite
I/native: I20230226 21:25:48.764724 10605 resource_util_android.cc:89] Successfully loaded: hand_recrop.tflite
I/native: I20230226 21:25:48.764724 10599 resource_util_android.cc:89] Successfully loaded: hand_recrop.tflite
I/tflite: Created TensorFlow Lite XNNPACK delegate for CPU.
I/tflite: Replacing 162 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 2 partitions.
E/OperationsUtils: NN_RET_CHECK failed (frameworks/ml/nn/common/OperationsUtils.cpp:43): type == expectedTypes[i] Invalid input tensor type TENSOR_FLOAT16 for input 1, expected TENSOR_FLOAT32
E/Utils: Validation failed for operation CONV_2D
E/tflite: NN API returned error ANEURALNETWORKS_BAD_DATA at line 1137 while adding operation.
E/tflite: Restored original execution plan after delegate application failure.
I/tflite: Replacing 63 node(s) with delegate (TfLiteNnapiDelegate) node, yielding 1 partitions.
E/FrameProcessor: Mediapipe error: 
    com.google.mediapipe.framework.MediaPipeException: internal: Graph has errors: 
    Calculator::Open() for node "holisticlandmarkcpu__facelandmarksfromposecpu__facelandmarkcpu__inferencecalculator__holisticlandmarkcpu__facelandmarksfromposecpu__facelandmarkcpu__InferenceCalculator" failed: ; (interpreter_->ModifyGraphWithDelegate(delegate_.get()))==(kTfLiteOk)cpu.cc:217) 
        at com.google.mediapipe.framework.Graph.nativeMovePacketToInputStream(Native Method)
        at com.google.mediapipe.framework.Graph.addConsumablePacketToInputStream(Graph.java:395)
        at com.google.mediapipe.components.FrameProcessor.onNewFrame(FrameProcessor.java:458)
        at com.google.mediapipe.components.ExternalTextureConverter$RenderThread.renderNext(ExternalTextureConverter.java:403)
        at com.google.mediapipe.components.ExternalTextureConverter$RenderThread.lambda$onFrameAvailable$0$ExternalTextureConverter$RenderThread(ExternalTextureConverter.java:338)
        at com.google.mediapipe.components.-$$Lambda$ExternalTextureConverter$RenderThread$IXJjtE68JMV2CMJNQcFYhTUqhj8.run(Unknown Source:4)
        at android.os.Handler.handleCallback(Handler.java:938)
        at android.os.Handler.dispatchMessage(Handler.java:99)
        at android.os.Looper.loop(Looper.java:223)
        at com.google.mediapipe.glutil.GlThread.run(GlThread.java:141)
I/tflite: Replacing 63 node(s) with delegate (TfLiteNnapiDelegate) node, yielding 1 partitions.

System information

OS Platform and Distribution : Ubuntu 18.04.6 LTS, Android 11): Compiler version : gcc 11.1.0 Programming Language and version : Mediapipe (C++, Python 3.6, Java) MediaPipe version: v0.8.9 Bazel version: 5.2.0 Android SDK and NDK versions (if android): NDK 21.0.6113669, Android SDK Platform 11 OpenCV version (if running on desktop): 3.4.19

ayushgdev commented 1 year ago

Hello @taeyeonlee Thanks for bringing in this issue. Please allow us some time to go through the issue and replicate at our end. Meanwhile, would you try this in the latest mediapipe v0.9.1 and let us know if the results are the same?

taeyeonlee commented 1 year ago

Hello @ayushgdev Thank you for time. In the latest mediapipe v0.9.1, It's working well with the modified face_landmark_cpu.pbtxt,
but, in the latest mediapipe v0.9.1, when building the mediapipe Holistic AAR, I modified the hand_recrop_by_roi_cpu.pbtxt, like below, to use nnapi.

# Predicts hand re-crop rectangle.
node {
  calculator: "InferenceCalculator"
  input_stream: "TENSORS:initial_crop_tensor"
  output_stream: "TENSORS:landmark_tensors"
  options: {
    [mediapipe.InferenceCalculatorOptions.ext] {
      model_path: "mediapipe/modules/holistic_landmark/hand_recrop.tflite"
      delegate { nnapi {} }
    }
  }
}

then, the Android app runs following error. 

02/27 21:55:42: Launching 'app' 
Install successfully finished in 4 s 925 ms.
$ adb shell am start -n "com.lite.holistic_tracking/com.lite.holistic_tracking.MainActivity" -a android.intent.action.MAIN -c android.intent.category.LAUNCHER
Connected to process 32480 on device 
Capturing and displaying logcat messages from application. This behavior can be disabled in the "Logcat output" section of the "Debugger" settings page.
I/Typeface: Current DefaultTypeface ( default_lgsmartui )
D/NetworkSecurityConfig: No Network Security Config specified, using platform default
D/NetworkSecurityConfig: No Network Security Config specified, using platform default
W/listic_trackin: Accessing hidden method Landroid/view/View;->computeFitSystemWindows(Landroid/graphics/Rect;Landroid/graphics/Rect;)Z (greylist, reflection, allowed)
W/listic_trackin: Accessing hidden method Landroid/view/ViewGroup;->makeOptionalFitsSystemWindows()V (greylist, reflection, allowed)
D/PermissionHelper: checkAndRequestCameraPermissions
V/holistic_Service: [taeyeon] onCreate 
I/native: I20230227 21:55:49.965055 32480 asset_manager_util.cc:59] Created global reference to asset manager.
I/AdrenoGLES-0: QUALCOMM build                   : e5dfe77, I46bb81ccf7
    Build Date                       : 08/16/20
    OpenGL ES Shader Compiler Version: EV031.32.02.00
    Local Branch                     : 
    Remote Branch                    : 
    Remote Branch                    : 
    Reconstruct Branch               : 
I/AdrenoGLES-0: Build Config                     : S P 10.0.5 AArch64
I/AdrenoGLES-0: Driver Path                      : /vendor/lib64/egl/libGLESv2_adreno.so
I/AdrenoGLES-0: PFP: 0x016dd091, ME: 0x00000000
E/libEGL: validate_display:87 error 3008 (EGL_BAD_DISPLAY)
I/native: I20230227 21:55:50.026330 32480 gl_context_egl.cc:84] Successfully initialized EGL. Major : 1 Minor: 5
I/native: I20230227 21:55:50.028769 32533 gl_context.cc:342] GL version: 3.2 (OpenGL ES 3.2 V@0502.0 (GIT@e5dfe77, I46bb81ccf7, 1597647511) (Date:08/16/20))
D/GlThread: Starting GL thread ExternalTextureConverter
I/holistic_Service: [taeyeon] startCamera()
I/CameraManagerGlobal: Connecting to camera service
I/CameraManagerGlobal: boundary_number_of_camera = 3
I/chatty: uid=10353(com.lite.holistic_tracking) identical 2 lines
I/CameraManagerGlobal: boundary_number_of_camera = 3
W/CameraManagerGlobal: [soar.cts] ignore the status update of camera: 3
I/CameraManagerGlobal: boundary_number_of_camera = 3
W/CameraManagerGlobal: [soar.cts] ignore the status update of camera: 4
I/CameraManagerGlobal: boundary_number_of_camera = 3
W/CameraManagerGlobal: [soar.cts] ignore the status update of camera: 5
I/CameraManagerGlobal: boundary_number_of_camera = 3
W/CameraManagerGlobal: [soar.cts] ignore the status update of camera: 6
I/CameraManagerGlobal: boundary_number_of_camera = 3
W/CameraManagerGlobal: [soar.cts] ignore the status update of camera: 7
I/CameraManagerGlobal: boundary_number_of_camera = 3
W/CameraManagerGlobal: [soar.cts] ignore the status update of camera: 8
I/CameraManagerGlobal: boundary_number_of_camera = 3
W/CameraManagerGlobal: [soar.cts] ignore the status update of camera: 9
I/CameraManagerGlobal: boundary_number_of_camera = 3
I/CameraManager: camera Id = 0, packageName = com.lite.holistic_tracking
I/CameraManagerGlobal: boundary_number_of_camera = 3
I/CameraManagerGlobal: boundary_number_of_camera = 3
I/CameraManagerGlobal: boundary_number_of_camera = 3
W/CameraManagerGlobal: ignore the torch status update of camera: 5
I/CameraManager: camera Id = 0, packageName = com.lite.holistic_tracking
I/CameraManager: camera Id = 1, packageName = com.lite.holistic_tracking
I/CameraManager: camera Id = 2, packageName = com.lite.holistic_tracking
I/CameraManagerGlobal: boundary_number_of_camera = 3
I/CameraManagerGlobal: boundary_number_of_camera = 3
D/CameraRepository: Added camera: 0
I/CameraManagerGlobal: boundary_number_of_camera = 3
I/CameraManager: camera Id = 0, packageName = com.lite.holistic_tracking
D/CameraXPreviewHelper: Camera target size ratio: 0.562500 width: 1080
D/CameraXPreviewHelper: Camera size candidate width: 4000 height: 3000 ratio: 1.333333 cost: 14400.000000
D/CameraXPreviewHelper: Camera size candidate width: 4000 height: 2250 ratio: 1.777778 cost: 15253.333333
D/CameraXPreviewHelper: Camera size candidate width: 4000 height: 1760 ratio: 2.272727 cost: 16203.636364
D/CameraXPreviewHelper: Camera size candidate width: 3840 height: 2160 ratio: 1.777778 cost: 15093.333333
D/CameraXPreviewHelper: Camera size candidate width: 3264 height: 2448 ratio: 1.333333 cost: 13664.000000
D/CameraXPreviewHelper: Camera size candidate width: 3264 height: 1836 ratio: 1.777778 cost: 14517.333333
D/CameraXPreviewHelper: Camera size candidate width: 3264 height: 1434 ratio: 2.276151 cost: 15474.209205
D/CameraXPreviewHelper: Camera size candidate width: 2560 height: 1920 ratio: 1.333333 cost: 12960.000000
D/CameraXPreviewHelper: Camera size candidate width: 2048 height: 1536 ratio: 1.333333 cost: 12448.000000
D/CameraXPreviewHelper: Camera size candidate width: 1920 height: 1080 ratio: 1.777778 cost: 13173.333333
D/CameraXPreviewHelper: Camera size candidate width: 1640 height: 720 ratio: 2.277778 cost: 13853.333333
D/CameraXPreviewHelper: Camera size candidate width: 1440 height: 1080 ratio: 1.333333 cost: 11840.000000
D/CameraXPreviewHelper: Camera size candidate width: 1440 height: 960 ratio: 1.500000 cost: 12160.000000
D/CameraXPreviewHelper: Camera size candidate width: 1408 height: 1152 ratio: 1.222222 cost: 11594.666667
D/CameraXPreviewHelper: Camera size candidate width: 1280 height: 960 ratio: 1.333333 cost: 11680.000000
D/CameraXPreviewHelper: Camera size candidate width: 1280 height: 768 ratio: 1.666667 cost: 12320.000000
D/CameraXPreviewHelper: Camera size candidate width: 1280 height: 720 ratio: 1.777778 cost: 12533.333333
D/CameraXPreviewHelper: Camera size candidate width: 960 height: 720 ratio: 1.333333 cost: 11600.000000
D/CameraXPreviewHelper: Camera size candidate width: 960 height: 540 ratio: 1.777778 cost: 12453.333333
D/CameraXPreviewHelper: Camera size candidate width: 720 height: 720 ratio: 1.000000 cost: 11200.000000
D/CameraXPreviewHelper: Camera size candidate width: 720 height: 540 ratio: 1.333333 cost: 11840.000000
D/CameraXPreviewHelper: Camera size candidate width: 720 height: 480 ratio: 1.500000 cost: 12160.000000
D/CameraXPreviewHelper: Camera size candidate width: 640 height: 480 ratio: 1.333333 cost: 11920.000000
D/CameraXPreviewHelper: Camera size candidate width: 480 height: 640 ratio: 0.750000 cost: 600.000000
D/CameraXPreviewHelper: Camera size candidate width: 352 height: 288 ratio: 1.222222 cost: 11994.666667
D/CameraXPreviewHelper: Camera size candidate width: 320 height: 240 ratio: 1.333333 cost: 12240.000000
I/Camera2CameraInfo: Device Level: INFO_SUPPORTED_HARDWARE_LEVEL_3
D/CameraXPreviewHelper: Camera size candidate width: 176 height: 144 ratio: 1.222222 cost: 12170.666667
D/CameraRepository: Added camera: 1
I/CameraManagerGlobal: boundary_number_of_camera = 3
D/CameraXPreviewHelper: Optimal camera size width: 480 height: 640
I/CameraManager: camera Id = 1, packageName = com.lite.holistic_tracking
D/UseCaseAttachState: Active and attached use case: [] for camera: 0
I/Camera2CameraInfo: Device Level: INFO_SUPPORTED_HARDWARE_LEVEL_3
D/CameraRepository: Added camera: 2
I/CameraManagerGlobal: boundary_number_of_camera = 3
I/CameraManager: camera Id = 2, packageName = com.lite.holistic_tracking
I/Camera2CameraInfo: Device Level: INFO_SUPPORTED_HARDWARE_LEVEL_3
D/UseCaseAttachState: Active and attached use case: [] for camera: 2
D/UseCaseAttachState: Active and attached use case: [] for camera: 1
D/CameraOrientationUtil: getRelativeImageRotation: destRotationDegrees=0, sourceRotationDegrees=90, isOppositeFacing=true, result=90
D/DeferrableSurface: Surface created[total_surfaces=1, used_surfaces=0](androidx.camera.core.SurfaceRequest$2@82a0e0e}
D/CameraOrientationUtil: getRelativeImageRotation: destRotationDegrees=0, sourceRotationDegrees=90, isOppositeFacing=true, result=90
D/CameraXPreviewHelper: Received surface request for resolution 480x640
D/UseCaseAttachState: Active and attached use case: [] for camera: 0
D/CameraXPreviewHelper: Providing surface
D/UseCaseAttachState: All use case: [androidx.camera.core.Preview-c3567f32-f4a3-4818-90fa-115bf18d53ae66914771] for camera: 0
D/UseCaseAttachState: Active and attached use case: [androidx.camera.core.Preview-c3567f32-f4a3-4818-90fa-115bf18d53ae66914771] for camera: 0
D/GlThread: Starting GL thread ExternalTextureConverter
D/CameraStateRegistry: tryOpenCamera(Camera@a78ccd6[id=0]) [Available Cameras: 1, Already Open: false (Previous state: null)] --> SUCCESS
D/CameraStateRegistry: Recalculating open cameras:
    Camera                                       State                 
    -------------------------------------------------------------------
    Camera@aa595f3[id=1]                         UNKNOWN               
    Camera@e799e4f[id=2]                         UNKNOWN               
    Camera@a78ccd6[id=0]                         OPENING               
    -------------------------------------------------------------------
    Open count: 1 (Max allowed: 1)
D/UseCaseAttachState: All use case: [androidx.camera.core.Preview-c3567f32-f4a3-4818-90fa-115bf18d53ae66914771] for camera: 0
I/CameraManager: camera Id = 0, packageName = com.lite.holistic_tracking
I/CameraManagerGlobal: boundary_number_of_camera = 3
E/CameraManagerGlobal: Camera 6 is not available. Ignore physical camera status change
E/CameraManagerGlobal: Camera 7 is not available. Ignore physical camera status change
E/CameraManagerGlobal: Camera 8 is not available. Ignore physical camera status change
E/CameraManagerGlobal: Camera 9 is not available. Ignore physical camera status change
I/CameraManagerGlobal: boundary_number_of_camera = 3
I/CameraManagerGlobal: boundary_number_of_camera = 3
W/CameraManagerGlobal: ignore the torch status update of camera: 5
D/UseCaseAttachState: Active and attached use case: [androidx.camera.core.Preview-c3567f32-f4a3-4818-90fa-115bf18d53ae66914771] for camera: 0
D/CameraStateRegistry: Recalculating open cameras:
    Camera                                       State                 
    -------------------------------------------------------------------
    Camera@aa595f3[id=1]                         UNKNOWN               
    Camera@e799e4f[id=2]                         UNKNOWN               
    Camera@a78ccd6[id=0]                         OPEN                  
    -------------------------------------------------------------------
    Open count: 1 (Max allowed: 1)
D/UseCaseAttachState: All use case: [androidx.camera.core.Preview-c3567f32-f4a3-4818-90fa-115bf18d53ae66914771] for camera: 0
D/DeferrableSurface: New surface in use[total_surfaces=1, used_surfaces=1](androidx.camera.core.SurfaceRequest$2@82a0e0e}
D/DeferrableSurface: use count+1, useCount=1 androidx.camera.core.SurfaceRequest$2@82a0e0e
D/CaptureSession: Opening capture session.
D/CaptureSession: Attempting to send capture request onConfigured
D/CaptureSession: Issuing request for session.
D/CaptureSession: CameraCaptureSession.onConfigured() mState=OPENED
D/CaptureSession: CameraCaptureSession.onReady() OPENED
D/ExternalTextureConv: Created output texture: 4 width: 1920 height: 1080
I/native: I20230227 21:55:50.738000 32549 graph.cc:476] Start running the graph, waiting for inputs.
I/native: I20230227 21:55:50.738174 32549 gl_context_egl.cc:84] Successfully initialized EGL. Major : 1 Minor: 5
I/native: I20230227 21:55:50.739094 32569 gl_context.cc:342] GL version: 3.2 (OpenGL ES 3.2 V@0502.0 (GIT@e5dfe77, I46bb81ccf7, 1597647511) (Date:08/16/20))
I/native: I20230227 21:55:50.739326 32549 gl_context_egl.cc:84] Successfully initialized EGL. Major : 1 Minor: 5
I/native: I20230227 21:55:50.740880 32570 gl_context.cc:342] GL version: 3.2 (OpenGL ES 3.2 V@0502.0 (GIT@e5dfe77, I46bb81ccf7, 1597647511) (Date:08/16/20))
I/native: I20230227 21:55:50.741107 32549 gl_context_egl.cc:84] Successfully initialized EGL. Major : 1 Minor: 5
I/native: I20230227 21:55:50.742739 32571 gl_context.cc:342] GL version: 3.2 (OpenGL ES 3.2 V@0502.0 (GIT@e5dfe77, I46bb81ccf7, 1597647511) (Date:08/16/20))
D/ExternalTextureConv: Created output texture: 5 width: 1920 height: 1080
I/native: I20230227 21:55:50.756145 32563 resource_util_android.cc:90] Successfully loaded: hand_landmark_lite.tflite
D/ExternalTextureConv: Created output texture: 6 width: 1920 height: 1080
I/native: I20230227 21:55:50.762962 32564 resource_util_android.cc:90] Successfully loaded: hand_landmark_lite.tflite
I/native: I20230227 21:55:50.764577 32562 resource_util_android.cc:90] Successfully loaded: face_landmark_with_attention.tflite
I/native: I20230227 21:55:50.768643 32561 resource_util_android.cc:90] Successfully loaded: pose_landmark_lite.tflite
I/native: I20230227 21:55:50.769606 32565 resource_util_android.cc:90] Successfully loaded: pose_detection.tflite
I/native: I20230227 21:55:50.776342 32564 jni_util.cc:41] GetEnv: not attached
I/native: I20230227 21:55:50.781709 32569 jni_util.cc:41] GetEnv: not attached
I/tflite: Created TensorFlow Lite delegate for NNAPI.
I/tflite: Initialized TensorFlow Lite runtime.
I/native: I20230227 21:55:50.787715 32561 jni_util.cc:41] GetEnv: not attached
I/Manager: DeviceManager::DeviceManager
I/Manager: findAvailableDevices
I/Manager: Found interface qti-default
I/VersionedInterfaces: Capab {.relaxedFloat32toFloat16PerformanceScalar = {.execTime = 0.600000, .powerUsage = 0.800000}, .relaxedFloat32toFloat16PerformanceTensor = {.execTime = 0.600000, .powerUsage = 0.800000}, .operandPerformance = [15]{{.type = FLOAT32, .info = {.execTime = 0.600000, .powerUsage = 0.800000}}, {.type = INT32, .info = {.execTime = 0.600000, .powerUsage = 0.800000}}, {.type = UINT32, .info = {.execTime = 0.600000, .powerUsage = 0.800000}}, {.type = TENSOR_FLOAT32, .info = {.execTime = 0.600000, .powerUsage = 0.800000}}, {.type = TENSOR_INT32, .info = {.execTime = 0.600000, .powerUsage = 0.800000}}, {.type = TENSOR_QUANT8_ASYMM, .info = {.execTime = 0.700000, .powerUsage = 0.700000}}, {.type = BOOL, .info = {.execTime = 0.600000, .powerUsage = 0.800000}}, {.type = TENSOR_QUANT16_SYMM, .info = {.execTime = 0.700000, .powerUsage = 0.700000}}, {.type = TENSOR_FLOAT16, .info = {.execTime = 0.600000, .powerUsage = 0.800000}}, {.type = TENSOR_BOOL8, .info = {.execTime = 0.600000, .powerUsage = 0.800000}}, {.type = FLOAT16, .info = {.execTime = 0.600000, .powerUsage = 0.800000}}, {.type = TENSOR_QUANT8_SYMM_PER_CHANNEL, .info = {.execTime = 0.700000, .powerUsage = 0.700000}}, {.type = TENSOR_QUANT16_ASYMM, .info = {.execTime = 0.700000, .powerUsage = 0.700000}}, {.type = TENSOR_QUANT8_SYMM, .info = {.execTime = 0.700000, .powerUsage = 0.700000}}, {.type = TENSOR_QUANT8_ASYMM_SIGNED, .info = {.execTime = 0.700000, .powerUsage = 0.700000}}}, .ifPerformance = {.execTime = 0.600000, .powerUsage = 0.800000}, .whilePerformance = {.execTime = 0.600000, .powerUsage = 0.800000}}
I/Manager: Found interface qti-dsp
I/VersionedInterfaces: Capab {.relaxedFloat32toFloat16PerformanceScalar = {.execTime = 2.000000, .powerUsage = 2.000000}, .relaxedFloat32toFloat16PerformanceTensor = {.execTime = 2.000000, .powerUsage = 2.000000}, .operandPerformance = [15]{{.type = FLOAT32, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = INT32, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = UINT32, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_FLOAT32, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_INT32, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_QUANT8_ASYMM, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = BOOL, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_QUANT16_SYMM, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_FLOAT16, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_BOOL8, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = FLOAT16, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_QUANT8_SYMM_PER_CHANNEL, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_QUANT16_ASYMM, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_QUANT8_SYMM, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_QUANT8_ASYMM_SIGNED, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}}, .ifPerformance = {.execTime = 2.000000, .powerUsage = 2.000000}, .whilePerformance = {.execTime = 2.000000, .powerUsage = 2.000000}}
I/Manager: Found interface qti-gpu
I/VersionedInterfaces: Capab {.relaxedFloat32toFloat16PerformanceScalar = {.execTime = 1.100000, .powerUsage = 1.100000}, .relaxedFloat32toFloat16PerformanceTensor = {.execTime = 1.100000, .powerUsage = 1.100000}, .operandPerformance = [15]{{.type = FLOAT32, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = INT32, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = UINT32, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_FLOAT32, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_INT32, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_QUANT8_ASYMM, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = BOOL, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_QUANT16_SYMM, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_FLOAT16, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_BOOL8, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = FLOAT16, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_QUANT8_SYMM_PER_CHANNEL, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_QUANT16_ASYMM, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_QUANT8_SYMM, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_QUANT8_ASYMM_SIGNED, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}}, .ifPerformance = {.execTime = 1.100000, .powerUsage = 1.100000}, .whilePerformance = {.execTime = 1.100000, .powerUsage = 1.100000}}
I/Manager: Found interface qti-hta
I/VersionedInterfaces: Capab {.relaxedFloat32toFloat16PerformanceScalar = {.execTime = 2.000000, .powerUsage = 2.000000}, .relaxedFloat32toFloat16PerformanceTensor = {.execTime = 2.000000, .powerUsage = 2.000000}, .operandPerformance = [15]{{.type = FLOAT32, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = INT32, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = UINT32, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_FLOAT32, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_INT32, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_QUANT8_ASYMM, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = BOOL, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_QUANT16_SYMM, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_FLOAT16, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_BOOL8, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = FLOAT16, .info = {.execTime = 2.000000, .powerUsage = 2.000000}}, {.type = TENSOR_QUANT8_SYMM_PER_CHANNEL, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_QUANT16_ASYMM, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_QUANT8_SYMM, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}, {.type = TENSOR_QUANT8_ASYMM_SIGNED, .info = {.execTime = 1.100000, .powerUsage = 1.100000}}}, .ifPerformance = {.execTime = 2.000000, .powerUsage = 2.000000}, .whilePerformance = {.execTime = 2.000000, .powerUsage = 2.000000}}
W/tflite: NNAPI SL driver did not implement SL_ANeuralNetworksDiagnostic_registerCallbacks!
I/TypeManager: Failed to read /vendor/etc/nnapi_extensions_app_allowlist ; No app allowlisted for vendor extensions use.
I/native: I20230227 21:55:50.819396 32562 jni_util.cc:41] GetEnv: not attached
I/native: I20230227 21:55:50.854482 32566 jni_util.cc:41] GetEnv: not attached
I/native: I20230227 21:55:50.884887 32568 jni_util.cc:41] GetEnv: not attached
I/native: I20230227 21:55:50.922266 32563 jni_util.cc:41] GetEnv: not attached
W/tflite: NNAPI SL driver did not implement SL_ANeuralNetworksDiagnostic_registerCallbacks!
I/native: I20230227 21:55:51.108714 32567 jni_util.cc:41] GetEnv: not attached
I/native: I20230227 21:55:51.134300 32565 resource_util_android.cc:90] Successfully loaded: hand_recrop.tflite
E/tflite: The model is not a valid Flatbuffer buffer
I/native: I20230227 21:55:51.134805 32561 resource_util_android.cc:90] Successfully loaded: hand_recrop.tflite
I/native: I20230227 21:55:51.135450 32566 resource_util_android.cc:90] Successfully loaded: face_detection_short_range.tflite
W/tflite: NNAPI SL driver did not implement SL_ANeuralNetworksDiagnostic_registerCallbacks!
W/tflite: NNAPI SL driver did not implement SL_ANeuralNetworksDiagnostic_registerCallbacks!
E/FrameProcessor: Mediapipe error: 
    com.google.mediapipe.framework.MediaPipeException: internal: Graph has errors: 
    Calculator::Open() for node "holisticlandmarkcpu__handlandmarksleftandrightcpu__handlandmarksfromposecpu_1__handrecropbyroicpu__inferencecalculator__holisticlandmarkcpu__handlandmarksleftandrightcpu__handlandmarksfromposecpu_1__handrecropbyroicpu__InferenceCalculator" failed: ; RET_CHECK failure (mediapipe/util/tflite/tflite_model_loader.cc:41) modelFailed to load model from path mediapipe/modules/holistic_landmark/hand_recrop.tflite
        at com.google.mediapipe.framework.Graph.nativeMovePacketToInputStream(Native Method)
        at com.google.mediapipe.framework.Graph.addConsumablePacketToInputStream(Graph.java:407)
        at com.google.mediapipe.components.FrameProcessor.onNewFrame(FrameProcessor.java:458)
        at com.google.mediapipe.components.ExternalTextureConverter$RenderThread.renderNext(ExternalTextureConverter.java:485)
        at com.google.mediapipe.components.ExternalTextureConverter$RenderThread.lambda$onFrameAvailable$0$ExternalTextureConverter$RenderThread(ExternalTextureConverter.java:413)
        at com.google.mediapipe.components.-$$Lambda$ExternalTextureConverter$RenderThread$IXJjtE68JMV2CMJNQcFYhTUqhj8.run(Unknown Source:4)
        at android.os.Handler.handleCallback(Handler.java:938)
        at android.os.Handler.dispatchMessage(Handler.java:99)
        at android.os.Looper.loop(Looper.java:223)
        at com.google.mediapipe.glutil.GlThread.run(GlThread.java:141)
I/tflite: Replacing 63 node(s) with delegate (TfLiteNnapiDelegate) node, yielding 1 partitions.

In the mediapipe v0.8.9, It's working well with the modified hand_recrop_by_roi_cpu.pbtxt (nnapi). In the latest mediapipe v0.9.1, It doesn't work with the modified hand_recrop_by_roi_cpu.pbtxt (nnapi). In the mediapipe v0.8.9, It doesn't work with the modified face_landmark_cpu.pbtxt (nnapi). In the latest mediapipe v0.9.1, It's working well with the modified face_landmark_cpu.pbtxt (nnapi).

ayushgdev commented 1 year ago

@hadon Can you help in this issue ?