Registration of template data on 3D point cloud raw scanned data

[khg2478]

Dear Author,

Hi. Thank you for your great work first of all. It is very useful model indeed.

I want to do some experiments with your data and wish to know how I can align a template mesh on a raw scanned pcd data.

In the paper, it mentions that the registration work was done by "D.A. Hirshberg, M. Loper, E. Rachlin, and M.J. Black. 2012. Coregistration: Simultaneous alignment and modeling of articulated 3D shape. In European Conference on Computer Vision. 242–255." but I am struggling to find a way to align those two pcd data.

I have tested some of existing non-rigid ICP source codes that are publicly available but they didn't help me.

Would you clarify more in detail?

Thanks

[khg2478]

Hello once again,

I know it is not directly related to your project but it will be very helpful for me and some other researchers too.

Would you give me some advice? Thanks

[TimoBolkart]

I want to do some experiments with your data and wish to know how I can align a template mesh on a raw scanned pcd data.

If you want to work with our face data, we make them available with registrations (see the FLAME website for details about the available data).

To fit the template to a mesh in arbitrary mesh topology is similar to the fit_3D_mesh but replacing the vertex to vertex distance (which requires point correspondence) with a general point to surface distance. Does that help?

I am working on releasing such a functionality in the near future, but due to the upcoming CVPR deadline, I am unable to do this now.

[khg2478]

fit_3D_mesh

This helped me a lot! Thank you so much. Good luck with your upcoming submission! Best,

[TimoBolkart]

We just added some demo code to fit FLAME to as scan of arbitrary topology in the flame-fitting repo.

[khg2478]

@TimoBolkart Thank you so much. This is great work and I saved a lot of time. 👍 I appreciate your help.

[khg2478]

@TimoBolkart Hello. I wanted to align an image with corresponding 3D mesh object and solved it by running solvePnP with detected 2D landmark points with corresponding 3D vertices on the mesh. However, the alignment isn't very accurate and I am looking for a way to improve its fitting performance. Could you give me some advice??

[TimoBolkart]

Can you maybe show some images to make it clear what you are trying to do? What is the quality of the 3D mesh that you have? Why does fitting that 3D mesh not solve the problem?

[khg2478]

Thank you so much for your reply once again! I think my question was somewhat unclear. I apologize for that.

I mean, I wanted to align a 3D mesh with a 2D image as shown below. In the work below, I ran dlib based 2D landmark prediction and used solvePnP for the alignment. However, as you can see, although the scanned object or template mesh is originally from the image, I found it hard to align them nicely. So, I wanted to ask your advice for the alignment. :)

[TimoBolkart]

This looks like one of the VOCA or CoMA subjects, no? Where does the image come from that you want to project into? Is that one of the scanner's camera images? If so, I suspect that the camera transformation you use is not right. Where do you get the camera parameters from?

[khg2478]

YES! It's from a VOCA subject, which is "voca/imgs/FaceTalk_170728_03272_TA/sentence01/sentence01.000001.26_C.jpg".

For the camera parameters setting, I used ordinary parameters as below.

size = img.shape focal_length = size[1] center = (size[1]/2, size[0]/2) camera_matrix = np.array( [[focal_length, 0, center[0]], [0, focal_length, center[1]], [0, 0, 1]], dtype = "double" )

Please also note that the shape of the image is (1200,1600,3).

As there is no transformation matrix saved in this project, I need to find a way to solve this and wish to get some advice from you! :) Thank you very very very much for your help!

[TimoBolkart]

We do have camera calibrations for all data. I will find them and make them available

[khg2478]

We do have camera calibrations for all data. I will find them and make them available

That will be beautiful!! I hope that the data contains camera calibrations for all different degrees too! I really really appreciate your help! Thank you so much!

[TimoBolkart]

Sorry, I forgot to forward the links. You find here the CoMA calibations and VOCA calibations. Does that help?

[khg2478]

Thank you so much! You are the most diligent and nicest researcher I have ever seen. I really really appreciate your help. :) Have a great day!

[TimoBolkart]

Thank you for the kind words. Have a great day!