Revisiting basic landmark position printing

[ksyao2002]

Hello, I'm trying to print the landmarks at each frame for the face mesh model just as a proof of concept that I can access these positions. I looked through https://github.com/homuler/MediaPipeUnityPlugin/issues/5, https://github.com/homuler/MediaPipeUnityPlugin/issues/23, https://github.com/homuler/MediaPipeUnityPlugin/issues/55, https://github.com/homuler/MediaPipeUnityPlugin/issues/43, and what I thought we do is put the below code in RenderOutput:

public override void RenderOutput(WebCamScreenController screenController, TextureFrame textureFrame) {
        graph.ObserveOutputStream("face_landmarks", LandmarksCallback).AssertOk();
        if (multiFaceLandmarksStreamPoller != null)
        {
            Debug.Log("Fetching landmarks");
            var landmarks = FetchNext(multiFaceLandmarksStreamPoller, multiFaceLandmarksPacket, "face_landmarks");
            Debug.Log(landmarks[0]);
        }
        else
        {
            Debug.Log("can't fetch landmarks");
        }
        var faceMeshValue = FetchNextFaceMeshValue();
    RenderAnnotation(screenController, faceMeshValue);

    screenController.DrawScreen(textureFrame);
  }

In this function I get error that LandmarksCallback cannot convert to CalculatorGraph.NativePacketCallback.

And I added the following method in the same file (FaceMeshGraphs.cs):

static IntPtr LandmarksCallback(IntPtr ptr)
    {
        using (var packet = new NormalizedLandmarkListVectorPacket(ptr, false)) //originally this was NormalizedLandmarksListVectorPacket (note the extra s in Landmark)
        {
            using (var landmarks = packet.Get())
            {
                // do something
            }
            return Status.Ok().mpPtr;
        }
    }

But I get that IntPtr cannot be found, and NormalizedLandmarkList must be implicitly convertilce to IDisposable. Please let me know if I'm on the right track or any fixes!

[Jatin-Pawar]

Hey @ksyao2002

Are you trying to see if we get landmarks for the face and what all data do we get?

I am also trying to do the same. I looking into this project to see what all data do we get from MediaPipe in this plugin. I am interested in 3D coordinates (x,y,z) of the face mesh 438 points so that I can use these to attach objects like glasses or something.

I would be happy to connect and discuss/share my findings and doubts.

Kind Regards ~ jk!ng

[homuler]

I'm sorry I can't show you working code soon, but I'll update sample graphs later. For now, please refer to InstantMotionTrackingGraph. This graph outputs debug logs and I hope you can get some ideas from it.

A callback is defined here... https://github.com/homuler/MediaPipeUnityPlugin/blob/2ba2cbc8813735ddfbed3abd2124ce9500efddf7/Assets/Mediapipe/Samples/Graphs/InstantMotionTracking/Scripts/InstantMotionTrackingGraph.cs#L90-L107

and it is passed to MediaPipe here. https://github.com/homuler/MediaPipeUnityPlugin/blob/2ba2cbc8813735ddfbed3abd2124ce9500efddf7/Assets/Mediapipe/Samples/Graphs/InstantMotionTracking/Scripts/InstantMotionTrackingGraph.cs#L57

[ksyao2002]

Thanks for the comment! Please let me know when you are able to get the sample code available :) BTW, I added using System to the top and it fixed the IntPtr issue, my code is now

[AOT.MonoPInvokeCallback(typeof(CalculatorGraph.NativePacketCallback))]
    static IntPtr LandmarksCallback(IntPtr packetPtr)
    {
        try
        {
            using (var packet = new NormalizedLandmarkListVectorPacket(packetPtr, false))
            {
                List<NormalizedLandmarkList> landmarks = packet.Get();
                Debug.Log("**************************************CALLBACK ");
                Debug.Log(landmarks[0]);
                Debug.Log("**************************************DONE ");
                return Status.Ok().mpPtr;
            }
        }
        catch (Exception e)
        {
            return Status.FailedPrecondition(e.ToString()).mpPtr;
        }
    }

public override void RenderOutput(WebCamScreenController screenController, TextureFrame textureFrame) {
    graph.ObserveOutputStream(multiFaceLandmarksStream, LandmarksCallback).AssertOk();
    var faceMeshValue = FetchNextFaceMeshValue();
    RenderAnnotation(screenController, faceMeshValue);

    screenController.DrawScreen(textureFrame);
  }

Which compiles but when I run face mesh it just crashes the app. Checking logcat didn't help as I didn't see any Unity Debug statements.

[ksyao2002]

Any updates anyone?

[homuler]

At least,

graph.ObserveOutputStream(multiFaceLandmarksStream, LandmarksCallback).AssertOk();

must not be called in RenderOuput. Please see https://github.com/homuler/MediaPipeUnityPlugin/issues/106#issuecomment-808176493 more carefully.

By the way, if you call FetchNextFaceMeshValue, face landmarks' positions are fetched (if not, RenderAnnotation cannot renders annotations), so you don't have to call CalculatorGraph#ObserveOutputStream and subscribe the outputs.

public override void RenderOutput(WebCamScreenController screenController, TextureFrame textureFrame) {
    var faceMeshValue = FetchNextFaceMeshValue();

    if (faceMeshValue.MultiFaceLandmarks.Count > 0) {
        var landmarks = faceMeshValue.MultiFaceLandmarks[0];
        // Debug.Log(landmarks);
    }
    RenderAnnotation(screenController, faceMeshValue);

    screenController.DrawScreen(textureFrame);
}

[ksyao2002]

Thanks for the comment! Just using your code for RenderOutput actually worked, there was no need for the callback function. Now it prints the landmarks every frame! Just to make sure, the elements in the landmark array would follow the ordering from the official mediapipe documentation, in other words, this picture?

[homuler]

Probably yes. See also https://github.com/homuler/MediaPipeUnityPlugin/blob/8421e659414eeb52d4bad0178774e398832ed26a/Packages/com.github.homuler.mediapipe/Runtime/Scripts/Annotation/FaceLandmarkListAnnotationController.cs .

[ksyao2002]

Yep it looks the same. Hope others can find this thread useful too. Closing for now :)