gengshan-y
commented
1 year ago Hi, it seems either the principal point or translation values are incorrect. It would be useful to check whether those values make sense.
Besides, I would suggest using the 2D-3D correspondence demo to debug.
It calls the kp_reproj() function and I think this is what you want.
Hello Gengshan,
Thank you! We made some progress on projecting the vertices from rest space to 2D space.
First, we get the rtk and kaug as follows:
rtk = batch['rtk'] .view(bs,-1,4,4).permute(1,0,2,3).reshape(-1,4,4) .to(self.device).float() kaug = batch['kaug'] .view(bs,-1,4).permute(1,0,2).reshape(-1,4) .to(self.device).float()
Then we calculate Rmat, Tmat, K as follows:
Rmat = rtk[:,:3,:3] Tmat = rtk[:,:3,3] Kmat = K2mat(rtk[:,3,:]) Kaug = K2inv(kaug) # p = Kaug Kmat P Kinv = Kmatinv(Kaug.matmul(Kmat)) K = mat2K(Kmatinv(Kinv))
Next, we get rts:
bs=batch['dataid'].shape[0] data_info = self.init_dataset() data_offset = data_info['offset'] frameid = batch['frameid'] .view(bs,-1).permute(1,0).reshape(-1).cpu() dataid = batch['dataid'] .view(bs,-1).permute(1,0).reshape(-1).cpu() embedid = frameid + data_offset[dataid.long()] frameid = frameid + data_offset[dataid.long()] frameid = frameid.to(self.device)
bone_rts = self.model.module.nerf_body_rts(frameid) rts_fw = correct_rest_pose(opts, bone_rts, bone_rts_rst) rts_fw = rts_fw.view(-1,B,12)# B,12 rmat=rts_fw[:,:,:9] rmat=rmat.view(-1,B,3,3) tmat= rts_fw[:,:,9:12] rts_fw = torch.cat([rmat,tmat[...,None]],-1) rts_fw = rts_fw.view(-1,B,3,4)
And use blend_skinning_chunk() -> obj_to_cam ->pinhole_cam to get the 2D projection. We get the following result
We observed the result from blend_skinning_chunk seems reasonable however the projection result as shown in the image above is not in the right co-ordinate. (We think the right projection should start from (0,0) and up to (64,64) instead of starting from some negative value in the image)
We guess the reason for the shift in the axis is because of the wrong K. Could you please advise on the right way to project the 3D to 2D? Additionally, can you please have a quick look at the codes we sent above and see if we are making any mistakes there?
Many Thanks!