Nose and foot coordinates using key-point R-CNN and DensePose.

[ItamarSafriel]

Hello everyone! Given an image, say, this, I am using the Key-Point R-CNN (from detectron2) to get the coordinates of the nose and either foot. Using this code, I get a 3D-reconstruction. DensePose changes the person's pose (as shown in the previous link). Using the 3D model and the key-points, how can I get the 3D coordinates of the nose and feet? (As of right now, they do not match). I used the projection of the model to the Y-Z plane then tried looking for the right and left most maxima but couldn't find a good enough algorithm.

I hope this is the correct place to post this, if not, please close this issue and if possible refer me to to correct place.

Thanks!

[frankkim1108]

why don't you try using densepose keypoints?

[ItamarSafriel]

why don't you try using densepose keypoints?

What do you mean? Are you talking about this model? Or does DensePose have 3D keypoints? Can you link it?

Thanks!

[frankkim1108]

@ItamarSafriel I was talking about COCO-Keypoints

[ItamarSafriel]

@ItamarSafriel I was talking about COCO-Keypoints

Well, I need to get the 3D keypoints, this is why I am using densepose. As I mentioned, after I create the 3D model the COCO-keypoints do not align.

Any ideas?

[vkhalidov]

Hi @ItamarSafriel , DensePose estimates a mapping from a generic 3D mesh to part-based 2D manifolds parameterized by U and V coordinates. So when you map the IUV values back to 3D, you get points on a generic mesh that are the same for all poses. One can map textures using this method, but cannot perform 3D reconstruction.

[hosnasattar]

if you load SMPL body model in MAYA you can select vertices ids, then you can use those ids to select dens pose keypoints. in the picture below I have my selections for example