Face Mesh solution Unity Local cords move opposite direction

onuralpszr commented 1 year ago

Plugin Version or Commit ID

v0.10.3

Unity Version

2021.3.3f1

Your Host OS

Fedora 37

Target Platform

Linux Standalone, Android, iOS

Description

I was experimenting how to use Face Mesh I found how to convert cords and get landmark in normalized and Local way but I notice a problem . I was trying to make Face Mask rig with vertices list. I was able to attach that mask into Unity Local cords system (converted from normalized landmark cords) but when I rotate my or move my head left for example mask goes right side. It is clearly act mirrored. I tried to product X with -1 to fix or try to play with Mirror mode but I was unable to fix that "movement" problem. How can I prevent that problem ?

Here is my snippet code

    public void UpdateFaceMesh(IList<NormalizedLandmarkList> target) {
     if (target != null)
     {
       var landmarkList = target[0];
       for (var i = 0; i < landmarkList.Landmark.Count - 10; i++)
       {
         var landmark = landmarkList.Landmark[i];
         vertextList[i] = annotation.NormalizedToLocalCord(landmark);
       }
       faceMesh.SetVertices(vertextList);
     }
    }

NormalizedToLocalCord function inside

  public Vector3 NormalizedToLocalCord(NormalizedLandmark target)
    {
      return GetScreenRect().GetPoint(target, RotationAngle.Rotation0, true);
    }

homuler commented 1 year ago

How about setting isMirrored to false?

return GetScreenRect().GetPoint(target, RotationAngle.Rotation0, false);

https://github.com/homuler/MediaPipeUnityPlugin/blob/1a8563095f0d5e6b9816a4f639728c2cbafab208/Packages/com.github.homuler.mediapipe/Runtime/Scripts/Unity/CoordinateSystem/ImageCoordinate.cs#L487-L489

onuralpszr commented 1 year ago

How about setting isMirrored to false?
return GetScreenRect().GetPoint(target, RotationAngle.Rotation0, false);
https://github.com/homuler/MediaPipeUnityPlugin/blob/1a8563095f0d5e6b9816a4f639728c2cbafab208/Packages/com.github.homuler.mediapipe/Runtime/Scripts/Unity/CoordinateSystem/ImageCoordinate.cs#L487-L489

I tried but did not worked. Let me re-try again.

onuralpszr commented 1 year ago

While I was testing I also changed this part for testing purpose

/Assets/MediaPipeUnity/Samples/Common/Scripts/GraphRunner.cs

    sidePacket.Emplace("input_horizontally_flipped", new BoolPacket(true)); // manually set to true // that make it worked.
   //sidePacket.Emplace("input_horizontally_flipped", new BoolPacket(inputHorizontallyFlipped)); // original

onuralpszr commented 1 year ago

How about setting isMirrored to false?
return GetScreenRect().GetPoint(target, RotationAngle.Rotation0, false);
https://github.com/homuler/MediaPipeUnityPlugin/blob/1a8563095f0d5e6b9816a4f639728c2cbafab208/Packages/com.github.homuler.mediapipe/Runtime/Scripts/Unity/CoordinateSystem/ImageCoordinate.cs#L487-L489

It did not worked. I used MP cannonical face mesh for control and It also rotate opposite direction (If I go left, mask goes right and I did not understand how to "scale" properly like when you put "face-mesh" with (green/red mesh) I was looking for how did you scale or resize properly. Because just giving vertex list not enough to control properly.

onuralpszr commented 1 year ago

Update : I gave main multi face landmark Gameobject as parent for fix scale and resize. Now It fits to scene and my face but If resolution is not cubic (For example If it is not something like 1000X1000 or 1200x1200 or 1345X1345) it will stretch and show ugly result. I was looking for how did you control that arrows. Can you give me pointers how to adapt different rotations and different resolutions. I can see that when I enable mesh they are change properly but not the mesh I put it in. I need 2 more parameters I guess.

homuler commented 1 year ago

If you don't know how to use Unity, please ask in the Unity forum or elsewhere. FYI: In the sample app, I implemented AutoFit by myself, but you may rather want to use AspectRationFitter or ContentSizeFitter.

I can see that when I enable mesh they are change properly but not the mesh I put it in. I need 2 more parameters I guess.

I'm sorry, I don't know what you mean here.

onuralpszr commented 1 year ago

If you don't know how to use Unity, please ask in the Unity forum or elsewhere. FYI: In the sample app, I implemented AutoFit by myself, but you may rather want to use AspectRationFitter or ContentSizeFitter.

I can see that when I enable mesh they are change properly but not the mesh I put it in. I need 2 more parameters I guess.

I'm sorry, I don't know what you mean here.

I fixed by changing GetPoint parameters to fit to screen properly.

but I will check those as well. Thank you.

But one thing I wonder when I tried on iOS, camera position on portrait it is still stay landscape. but landscape looks fine.

homuler commented 1 year ago

But one thing I wonder when I tried on iOS, camera position on portrait it is still stay landscape. but landscape looks fine.

Is this relevant to this plugin?

onuralpszr commented 1 year ago

But one thing I wonder when I tried on iOS, camera position on portrait it is still stay landscape. but landscape looks fine.

Is this relevant to this plugin?

It was and asked in here https://github.com/homuler/MediaPipeUnityPlugin/issues/847 and I got my answers as well. Thank you I am closing this issue as complete.

homuler / MediaPipeUnityPlugin