How to use Handtracking Android AAR in Unity

[taeyeonlee]

OS Platform and Distribution

Ubuntu 18.04.6 LTS (Handtracking Android AAR build), Windows (Wrapper AAR, Unity)

Compiler version

gcc 11.1.0

Programming Language and version

Mediapipe (C++, Python 3.6, Java), Unity (C#)

Installed using virtualenv? pip? Conda?(if python)

Python 3.6,

MediaPipe version

0.8.9

Bazel version

5.0

XCode and Tulsi versions (if iOS)

No response

Android SDK and NDK versions (if android)

NDK 21.0.6113669, Android SDK Platform 11

Android AAR (if android)

Yes

OpenCV version (if running on desktop)

No response

Describe the problem

I want to run Mediapipe handtracking AAR in Unity.
but, it runs following error.
Can you help to fix it ?

When using the below in the Wrapper AAR, Unity App runs the following error.
graph = new Graph();

Error Log
2023-03-07 15:50:43.664 24876-24906/com.DefaultCompany.HandtrackingForUnitydevelop E/Unity: AndroidJavaException: java.lang.NoClassDefFoundError: Failed resolution of: Lcom/google/common/flogger/FluentLogger;
    java.lang.NoClassDefFoundError: Failed resolution of: Lcom/google/common/flogger/FluentLogger;
        at com.google.mediapipe.framework.Graph.<clinit>(Graph.java:33)
        at com.rrrfffrrr.unity.mediapipe.handtracking.MediapipeGraph.<init>(MediapipeGraph.java:54)
        at com.unity3d.player.UnityPlayer.nativeRender(Native Method)
        at com.unity3d.player.UnityPlayer.access$300(Unknown Source:0)
        at com.unity3d.player.UnityPlayer$e$1.handleMessage(Unknown Source:95)
        at android.os.Handler.dispatchMessage(Handler.java:102)
        at android.os.Looper.loop(Looper.java:223)
        at com.unity3d.player.UnityPlayer$e.run(Unknown Source:20)
     Caused by: java.lang.ClassNotFoundException: Didn't find class "com.google.common.flogger.FluentLogger" on path: DexPathList[[zip file "/data/app/~~jLL-hFL3DI51XdORGxYeQA==/com.DefaultCompany.HandtrackingForUnitydevelop-ul3QLfEB0KT8b_G0z-uc3w==/base.apk"],nativeLibraryDirectories=[/data/app/~

Build Handtracking Android AAR (https://developers.google.com/mediapipe/framework/getting_started/android_archive_library)
-- Android studio --
Add gradle implement of previous built AAR
Make Java APIs that can be used in Unity3D
Build Wrapper AAR
-- Unity3D --
Create AndroidJavaObject
Add callback to java object
Send input (Texture.GetNativeTexturePtr().ToInt32())
Decode output and generate tracking data
You need add both Mediapipe AAR and Wrapper AAR at Assets/Plugins/Android/

Source : https://github.com/illisoft/HandtrackingForUnity/tree/develop

Complete Logs

Wrapper AAR
package com.rrrfffrrr.unity.mediapipe.handtracking;

import android.app.Activity;
import android.opengl.EGL14;
import android.util.Log;

import com.google.mediapipe.components.FrameProcessor;
import com.google.mediapipe.formats.proto.LandmarkProto;
import com.google.mediapipe.framework.*;
import com.google.mediapipe.glutil.EglManager;

import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.atomic.AtomicBoolean;

public class MediapipeGraph {

    private static final String TAG = "MediapipeGraph";
    private static final String BINARY_GRAPH_NAME = "hand_tracking_mobile_gpu.binarypb";
    private static final String INPUT_VIDEO_STREAM_NAME = "input_video";
    private static final String INPUT_NUM_HANDS_SIDE_PACKET_NAME = "num_hands" ;
    private static final String OUTPUT_VIDEO_STREAM_NAME = "output_video";
    private static final String OUTPUT_LANDMARKS_STREAM_NAME = "hand_landmarks"; 
    private static final int MAX_NUM_HANDS = 8;

    Boolean started = false;
    private static EglManager eglManager;
    private static Graph graph;
    PacketCreator packetCreator;
    protected FrameProcessor processor;
    Map<String, Packet> inputSidePackets = new HashMap<>();

    static {
        // Load all native libraries needed by the app.
        System.loadLibrary("mediapipe_jni");
        try {
            System.loadLibrary("opencv_java3");
        } catch (UnsatisfiedLinkError e) {
            // Some example apps (e.g. template matching) require OpenCV 4.
            System.loadLibrary("opencv_java4");
        }
    }

    public MediapipeGraph (Activity context,DataCallback callback, int numHands)
    {
        Log.i(TAG, "init ()");
        eglManager = new EglManager(EGL14.eglGetCurrentContext());
        AndroidAssetUtil.initializeNativeAssetManager(context);
        graph = new Graph();
      }
}

Unity
Assets\Mediapipe\Handtracking.cs
using System;
using UnityEngine;
using UnityEngine.Events;

namespace mediapipe.wrapper
{
    public class Handtracking : IDisposable
    {
        public readonly DataEvent OnData = new DataEvent();
        private AndroidJavaObject m_Instance;

        public Handtracking(int num_hands = 2) {
            AndroidJavaClass unityPlayer = new AndroidJavaClass("com.unity3d.player.UnityPlayer");
            AndroidJavaObject activity = unityPlayer.GetStatic<AndroidJavaObject>("currentActivity");
            m_Instance = new AndroidJavaObject("com.rrrfffrrr.unity.mediapipe.handtracking.MediapipeGraph", activity, new DataCallback(data: DataHandler), num_hands);
        }

\Assets\MediapipeAARTest.cs
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using mediapipe.wrapper;
using UnityEngine.Events;
using UnityEngine.Android;

public class MediapipeAARTest : MonoBehaviour
{
    public static Handtracking mediapipeAAR;

    protected WebCamTexture textureWebCam = null;

    void Start()
    {
        StartCamera();
        mediapipeAAR = new Handtracking(2);
        mediapipeAAR.OnData.AddListener(OnJavaCallback);
    }
}

[kuaashish]

Hi @taeyeonlee, We see that mediapipe does not have any such example officially published or available. Source mentioned here https://github.com/illisoft/HandtrackingForUnity/tree/develop belongs to the community repo and we recommend you to follow up and raise the issue with the mentioned repo. However, If you have any issue related to feature request, bugs, build/install or support. We are always take that here and help out with possible solution. Thank you!

[taeyeonlee]

Hi @kuaashish The repo is read-only, I can't raise the issue on the community repo. The community repo says that Unity App can use the mediapipe AAR using wrapper AAR. I think it is a good approach. I hope Google Mediapipe to support the approach officially.

[kuaashish]

@taeyeonlee, Thank you for understanding the concern, However, we have already this request #1369 running internally and will let you know if there will be any specific update on the request. Although, request you to follow up with above issue and please feel free to close this ticket. Thank you!

[taeyeonlee]

@kuaashish The request (#1369) is also what I want. But, in this issue (#4161), I want a quick support from an Expert like you to fix the error, when using AAR and Wrapper AAR in Unity Android.

Failed resolution of: Lcom/google/common/flogger/FluentLogger; 
        at com.google.mediapipe.framework.Graph.<clinit>(Graph.java:33)

[kuaashish]

@taeyeonlee, From the error log shared, Seems the issue is related to dependencies in the build/gradle file as similar error has been reported here, we would like to inspect the above file once. Further, flogger team can help more on the issue we suggest that raise a issue here. However, Unity is officially not supported by MediaPipe yet. This we are marking as feature request and sharing internally with team. Thank you!

[fadiaburaid]

@taeyeonlee for easier implementation of Mediapipe hand tracking on Android using Unity I suggest you use MediaPipe Unity Plugin. There are good examples to achieve the task.

[taeyeonlee]

@fadiaburaid Yes, the MediaPipeUnityPlugin is a good example for Unity. but Each Unity App must include the Mediapipe Solutions. It's duplicated. and the performance is lower than AAR. If Mediapipe Android AAR wrapper is possible, then mediapipe solution can run on background process. and All Unity Apps don't need to include the Mediapipe Solution and use it.

[fadiaburaid]

@taeyeonlee I am not fully versed in AAR. I just included the Mediapipe Unity Package in Packages folder and the models used in the streaming assets folder. Then I edited the Calculators Graph according to the solution I wanted to include. I managed to animate a full avatar on Android using Holistic solution and it is running fine on a non high-end phone.

https://user-images.githubusercontent.com/28487342/225882275-7b9d7ac7-1cb3-4de3-b7a7-b272b815bf78.mp4

[taeyeonlee]

@fadiaburaid Great, the performance of MediaPipeUnityPlugin is not bad. but it's lower than AAR.

[boostcat]

@fadiaburaid This effect is great, can you share the project code to the github platform? I have recently also used the holistic function of the mediapipeunityplugin to drive the limbs and fingers of the avatar, but there has been no progress. I hope to learn this part of your code. Thank you so much! !

@taeyeonlee I am not fully versed in AAR. I just included the Mediapipe Unity Package in Packages folder and the models used in the streaming assets folder. Then I edited the Calculators Graph according to the solution I wanted to include. I managed to animate a full avatar on Android using Holistic solution and it is running fine on a non high-end phone.

avatar.mp4