Cuboid to primitive cube

[emreboyraz0]

@homuler New version of plugin is very nice.

I am working on Objectron and I am trying to shoe 3d model shoes on foot.

Basically I created a primitive 3d cube and put 3d shoe model in it.

`if (runningMode == RunningMode.Sync) { var value = graphRunner.FetchNextValue();

                if (value.liftedObjects != null)
                {
                  Debug.Log("Annotations " + value.liftedObjects.Annotations);
                  //positioning
                    Cube.transform.position = new Vector3(value.liftedObjects.Annotations[0].Translation[0], value.liftedObjects.Annotations[0].Translation[1], value.liftedObjects.Annotations[0].Translation[2]);
                  //rotation
                    Vector3 forward;
                    forward.x = value.liftedObjects.Annotations[0].Rotation[0];
                    forward.y = value.liftedObjects.Annotations[0].Rotation[1];
                    forward.z = value.liftedObjects.Annotations[0].Rotation[2];
                    Vector3 upwards;
                    upwards.x = value.liftedObjects.Annotations[0].Rotation[3];
                    upwards.y = value.liftedObjects.Annotations[0].Rotation[4];
                    upwards.z = value.liftedObjects.Annotations[0].Rotation[5];
                    Quaternion lookOnLook = Quaternion.LookRotation(forward, upwards);
                    Cube.transform.rotation = lookOnLook;
                  //scaling
                  Cube.transform.localScale=new Vector3(value.liftedObjects.Annotations[0].Scale[0]*100,value.liftedObjects.Annotations[0].Scale[1]*100,value.liftedObjects.Annotations[0].Scale[2]*100);

                }

                liftedObjectsAnnotationController.DrawNow(value.liftedObjects);
                multiBoxRectsAnnotationController.DrawNow(value.multiBoxRects);
                multiBoxLandmarksAnnotationController.DrawNow(value.multiBoxLandmarks);
            }

`

I tried to change position, scale and rotation in ObjectronSolution.cs . But can not exactly achieve. Can you let me understand how can I achieve this? thanks

[homuler]

As for Annotation, CuboidAnnotation is the only one that I haven't been able to organize the code, so I haven't been able to cut out the process to helper functions. I'll implement those helper functions later under Mediapipe.Unity.CoordinateSystem, but you need to implement by yourself currently.

1. Position

   Cube.transform.position = new Vector3(value.liftedObjects.Annotations[0].Translation[0], value.liftedObjects.Annotations[0].Translation[1], value.liftedObjects.Annotations[0].Translation[2]);

   Use ObjectAnnotation#Keypoints (see here to know which is which) and CoordinateSystem.CameraCoordinate.GetLocalPosition instead. https://github.com/homuler/MediaPipeUnityPlugin/blob/0604681ed9cb41d5955709ffa7d537f20058ab00/Packages/com.github.homuler.mediapipe/Runtime/Scripts/Unity/CoordinateSystem/CameraCoordinate.cs#L34-L47

   You can use ObjectAnnotation#Translation, but probably it does not represent what you imagine. https://google.github.io/mediapipe/solutions/objectron.html#detected_objects

2. Scale Note that Camera Coordinate (coordinate system that ObjectAnnotation uses) is right-handed, but coordinate system in Unity is left-handed. https://google.github.io/mediapipe/solutions/objectron.html#camera-coordinate

   https://github.com/homuler/MediaPipeUnityPlugin/blob/0604681ed9cb41d5955709ffa7d537f20058ab00/Packages/com.github.homuler.mediapipe/Runtime/Scripts/Unity/Annotation/TransformAnnotation.cs#L67

3. Rotation This is probably the most complicated part. You should note 2 facts here.

   1. ObjectAnnotation#Rotation is row-major matrix.
   2. Camera Coordinate is right-handed.

   See also https://github.com/homuler/MediaPipeUnityPlugin/blob/0604681ed9cb41d5955709ffa7d537f20058ab00/Packages/com.github.homuler.mediapipe/Runtime/Scripts/Unity/Annotation/TransformAnnotation.cs#L44-L46

[emreboyraz0]

thanks for you reply. Previous version I got matrix4x4 to shoes pos, scale, rotation that rendered in Unity. But in IOS graph was not stable, app always crashed then I decided to use this version.

But in new version of plugins is too complicated for me. Even you explain can not figure it out like old version as you see in picture. .

[RICKYLIKEBIRD]

@homuler I tried to get the coordinates of a certain point as explained to you and put a ball, as shown in the figure But the position of the ball does not seem to be right My program Maybe I am missing something that needs attention?

[nenenkosi]

@homuler where do i get the RectTransform rectTransform from or what arguments do I need to pass for the rectTransform

https://github.com/homuler/MediaPipeUnityPlugin/blob/0604681ed9cb41d5955709ffa7d537f20058ab00/Packages/com.github.homuler.mediapipe/Runtime/Scripts/Unity/CoordinateSystem/CameraCoordinate.cs#L34-L47

[AbdallahKATAIE]

Hey @emreboyraz0 If I understood correctly, you were able to get Position, Scale, and Rotation correctly on the older version using matrix4x4 ?

I'm using this function to get Position, Scale, and Rotation by my rotations are really inaccurate.

[AOT.MonoPInvokeCallback(typeof(CalculatorGraph.NativePacketCallback))]
    static IntPtr MatrixCallback(IntPtr packetPtr)
    {
        try
      {
          using (var packet = new TimedModelMatrixProtoListPacket(packetPtr, false))
          {
                var matrixProtoList = packet.Get();
              if (matrixProtoList.ModelMatrix.Count > 0)
              {        
                     var matrix = Matrix4x4FromBytes(matrixProtoList.ModelMatrix[0].MatrixEntries);
                    Stablematrix = matrix;
                  }
          }

          //TODO: ensure the returned status won't be garbage collected prematurely.
          return Status.Ok().mpPtr;
      }
      catch (Exception e)
      {
          return Status.FailedPrecondition(e.ToString()).mpPtr;
      }
    }

Could you explain to me how you got Rotation to work well from Matrix4x4 in the older version ? This is my email if you prefer to message me privately : kataieabdallah@hotmail.com

[homuler]

@AbdallahKATAIE I'm sorry, but I'm not going to support older versions of the sample code, so if you plan to use the sample as-is, please try a newer version (Matrix4x4 is no longer used). Please let me know if you have any problems with the new version.

[edom18]

I have the same issue. I tried to use direction vectors that was mentioned by @homuler like below.

// var xDir = GetDirection(rotation[0], rotation[3], rotation[6], isXReversed, isYReversed, isInverted); 
var yDir = GetDirection(rotation[1], rotation[4], rotation[7], isXReversed, isYReversed, isInverted); 
var zDir = GetDirection(rotation[2], rotation[5], rotation[8], isXReversed, isYReversed, isInverted);

return Quaternion.LookRotation(zDir, yDir);

But my shoes still have wrong direction.

Should I do something to the vectors?

[homuler]

@edom18 I don't have time to check, so I'll just answer based on your comment. Maybe the zDir's are facing the other way? https://github.com/homuler/MediaPipeUnityPlugin/blob/0604681ed9cb41d5955709ffa7d537f20058ab00/Packages/com.github.homuler.mediapipe/Runtime/Scripts/Unity/Annotation/TransformAnnotation.cs#L44-L49

MediaPipe, I believe, assumes that the shoes are oriented with the toes facing forward, so please keep that in mind as well.

[edom18]

@homuler Thank you for your advice. I've tried it and I succeeded to fit my shoes!

[emreboyraz0]

@homuler Thank you for your advice. I've tried it and I succeeded to fit my shoes!

@edom18 could you share your code?

[H3NRILiN]

thanks for you reply. Previous version I got matrix4x4 to shoes pos, scale, rotation that rendered in Unity. But in IOS graph was not stable, app always crashed then I decided to use this version.

But in new version of plugins is too complicated for me. Even you explain can not figure it out like old version as you see in picture.

@emreboyraz0 Hi, I'm also making the try-on app, but encounting the low framerate issue and detection low accuracy problem Is posible that you can share the apk file? I want to compare your result to mine. Thanks

[homuler]

I'm sorry, I thought that the value of rotation was a rotation matrix, but it seems not.

rotation: rotation matrix from object coordinate frame to camera coordinate frame.

Although the documentation states that this matrix is a rotation matrix, it isn't an orthogonal matrix and non-orthogonal matrices are usually not called rotation matrices (see also https://google.github.io/mediapipe/solutions/objectron.html#ndc-space).

Therefore, the description of Rotation in https://github.com/homuler/MediaPipeUnityPlugin/issues/301#issuecomment-932098018 is incorrect. Also, as a consequence of this, it is not possible to place an object in Unity using the rotation value as it is.

[homuler]

See https://github.com/homuler/MediaPipeUnityPlugin/issues/237#issuecomment-1132355194 for information on how to get a Quaternion.

[AbdallahKATAIE]

I managed to get everything working, including the full try-on project, the Occlusion Shader fix, Rotation, Translation, Scale matrix etc... here's my email if you need them : kataieabdallah@hotmail.com

[emreboyraz0]

Hey @emreboyraz0 If I understood correctly, you were able to get Position, Scale, and Rotation correctly on the older version using matrix4x4 ?

I'm using this function to get Position, Scale, and Rotation by my rotations are really inaccurate.

[AOT.MonoPInvokeCallback(typeof(CalculatorGraph.NativePacketCallback))]
    static IntPtr MatrixCallback(IntPtr packetPtr)
    {
        try
      {
          using (var packet = new TimedModelMatrixProtoListPacket(packetPtr, false))
          {
                var matrixProtoList = packet.Get();
              if (matrixProtoList.ModelMatrix.Count > 0)
              {        
                     var matrix = Matrix4x4FromBytes(matrixProtoList.ModelMatrix[0].MatrixEntries);
                    Stablematrix = matrix;
                  }
          }

          //TODO: ensure the returned status won't be garbage collected prematurely.
          return Status.Ok().mpPtr;
      }
      catch (Exception e)
      {
          return Status.FailedPrecondition(e.ToString()).mpPtr;
      }
    }

Could you explain to me how you got Rotation to work well from Matrix4x4 in the older version ? This is my email if you prefer to message me privately : kataieabdallah@hotmail.com

Sorry for late answer. Old version is plugin has ram usage issues both android and iOS. I updated project to new version. Old version codes are removed from my computer. But you should get pos,scale,rotation from 4x4 matrix. I tried all combinations. But I said before ram usage of old version is not good and you can not continue to develop