How to distinguish between left and right feet in 3D objects detection

[z7212819]

Hello, now I can replace it with my own shoe model, but how do I distinguish the left and right feet so that the shoes can distinguish the left and right directions. As shown in the picture, the model showing the right foot on both feet is now displayed

[z7212819]

@sgowroji I'm glad you can reply to me. I am using linux. Currently, I only replace the bounding box with the model. What should I do to distinguish the left and right directions of the shoes?

[ahmadyan]

Hi, We don't provide distinction between the left and the right shoe in the model. You can always use a hack that the right shoe is the right of the left shoe.

[z7212819]

@ahmadyan I'm glad you can reply to me. As you said, the right shoe is the right shoe of the left shoe, but how can I tell the right foot that you are the other shoe of the left foot, because the orientation of the imported model is fixed. So my idea is how to display the orientation of the model of the other shoe through mirror translation after one shoe is displayed. If there is such a method, can you tell me how to do it? thank you

[ahmadyan]

One solution is to modify the graph, and create/compute the left_right bit in the AnnotatedFrames in here, Specifically add a left_right bit in the ObjectAnnotation proto here

then propagate this bit throughout the graph to the rendering calculator. In the rendering calculator you can flip the object if it is for the left shoe accordingly.

[z7212819]

@ahmadyan Hello, after I checked the code you provided, I found that the object will be ID numbered after it is detected. If I have two feet, the ID is 1 and 2, and I found that there is a calculator called switchcontainer, which can be used to distinguish the flag bits. So my idea is whether I can use switchcontainer to distinguish the ID, for example, when the ID is 1, it is the left foot, And then in contained_ Node, add annotations to model matrices calculator to modify the orientation of the model. If Id is 2, rotate the orientation. Is this feasible?

[RMiTiG]

@ahmadyan I have same question.I also use the hack to distinguish left and right,but it doesn't work when there is only one shoe in the image.And I believe that it could distinguish the left and right as it has ditinguish the front and back. But I don't understand how can I create/compute the left_right bit in the AnnotatedFrames

And I found that Objectron can get predicted segmentation mask from this page.Is it possible to distinguish the left and right by this mask? If it worked, how can I get this mask when I use two-stage pipeline? I just found that the box detections is created in ObjectDetectionOidV4Subgraph, and converted into rectangles in DetectionsToRectsCalculator.Thanks for any suggestions.

[z7212819]

@RMiTiG Hello, according to your description, can you tell the right foot from the left if you have two shoes? How do you do that? Thank you very much.

[RMiTiG]

@RMiTiG Hello, according to your description, can you tell the right foot from the left if you have two shoes? How do you do that? Thank you very much.

I just judge from their position.For example,a box with the center(1, 0, 0) is in the right of the box with the center(0, 0, 0) in right-handed coordinates.That is why it not works for one shoe.Two shoe maybe failed sometimes when the feet cross together.So I prefer to solve this problem in the graph.

[z7212819]

@RMiTiG I see, this method of yours is also very thoughtful. Although one foot cannot tell the position, at least you can tell the left and right feet. I think it is very impressive. I only need to identify the left and right feet in the shoe part, the other parts have been almost perfect, we can exchange various ideas about this solution. Which part of your scheme is modified to realize? If I find a method that can realize single foot recognition on the way, I would like to share this method with you. Thank you very much.

[RMiTiG]

Which part of your scheme is modified to realize

If you mean how can I realize the hack in rendering, I am sorry that I do not render by GlAnimationOverlayCalculator.I just export the object's pose as graph output and render my assets in other engine such as sceneform/scenekit.So it's easier for me to handle this.

One solution is to modify the graph, and create/compute the left_right bit in the AnnotatedFrames in here, Specifically add a left_right bit in the ObjectAnnotation proto here

then propagate this bit throughout the graph to the rendering calculator. In the rendering calculator you can flip the object if it is for the left shoe accordingly.

And I finally understand what's ahmadyan mean,this is the solution that you need I guess. I think we can realize the hack I said above in AnnotationsToModelMatricesCalculator, the lifted_objects includes the position we need.

[z7212819]

@RMiTiG Indeed, I use GlAnimationOverlayCalculator is rendered, as ahmadyan said need to create a left and right in the AnnotationsToModelMatricesCalculator flags, I don't have any idea how to do this. Lifted_objects is usually used for shoe rotation. My idea is to move the center of the Lifted_objects part to the center of the two feet and then put the render part to the left and right, but I don't know how to do this at the moment. What do you have in mind?

[RMiTiG]

My idea is to move the center of the Lifted_objects part to the center of the two feet and then put the render part to the left and right

Actually I don't get it because I don't know the workflow in GlAnimationOverlayCalculator.

I think we can realize the hack I said above in AnnotationsToModelMatricesCalculator, the lifted_objects includes the position we need.

My thought is you can change the model_matrices or mask_model_matrices which is determine the transform of model in GlAnimationOverlayCalculator. In my recognize,Lifted_objects contains a Annotation List.One annotation stands for one shoe, and we can distinguish left and right by comparing the annotation.Keypoints[0] which is stand for a box center when max_num_objects = 2.And we can do this in AnnotationsToModelMatricesCalculator that I marked above.As we know which one is right,we can flip the object by model_matrix.

Sorry that I can't test it and hope that it's useful for you.(It's sadly that the GlAnimationOverlayCalculator doesn't work in my iPhone)

[z7212819]

@RMiTiG Thank you very much for your answer, I will try later, if there is any breakthrough, I will tell you in this question, I am also trying to cross-platform research, if I can directly transplant the current solution to iOS, I will inform you in time, thank you very much

[ayushgdev]

Hello @z7212819

We are ending support for these MediaPipe Legacy Solutions, but upgrading the others. However, the libraries, documentation, and source code for all the MediapPipe Legacy Solutions will continue to be available in our GitHub repository and through library distribution services, such as Maven and NPM.

You can continue to use those legacy solutions in your applications if you choose. Though, we would request you to check new MediaPipe solutions which can help you more easily build and customize ML solutions for your applications. These new solutions will provide a superset of capabilities available in the legacy solutions.

[google-ml-butler[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

[google-ml-butler[bot]]

Closing as stale. Please reopen if you'd like to work on this further.

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No