search

chungyiweng / humannerf

HumanNeRF turns a monocular video of moving people into a 360 free-viewpoint video.
MIT License
786 stars 86 forks source link

Help regarding Incorrect Results on Custom Data #73

Closed Dipankar1997161 closed 10 months ago

Dipankar1997161 commented 1 year ago

Hello @chungyiweng ,

I downloaded the processed file from the repo drive https://github.com/Arthur151/ROMP/blob/master/docs/dataset.md#dataset-preparation

could you tell me - What could be the camera intrinsic and extrinsic values in case I want to provide them for 3d Reconstruction for ROMP based values

I could not find much information about cam_values over there.

In the processed file, I found these keys - dict_keys(['kp3d_mono', 'kp2d', 'kp3d', 'cam', 'poses', 'betas', 'trans']) cam and trans are 3x3 matrix but what exactly are they?? Not much info has been given

an example of the cam value: 'cam': array([[-2.7358444 , -0.12197854, -0.3725123 ], [-2.7359936 , -0.12215704, -0.37204784], [-2.736054 , -0.12224719, -0.37204236]], dtype=float32)

Poses used was poses[2] - I assumed its the poses parameters for kp3d(generated from GT 3d Keypoints)

The camera values I used: {'intrinsics': array([[443.4 , 0. , 514.9682 ], fx and fy are from ROMP config.py [ 0. , 443.4 , 501.88202], [ 0. , 0. , 1. ]], dtype=float32),

'extrinsics_ 1': array([[ 1. , 0. , 0. , -0.07911643], [ 0. , 1. , 0. , 0.42559049], [ 0. , 0. , 1. , 4.4544816 ], [ 0. , 0. , 0. , 1. ]]) 'extrinsics_ 2': array([[ 1. , 0. , 0. , 0.], [ 0. , 1. , 0. , 0.], [ 0. , 0. , 1. , 0. ], [ 0. , 0. , 0. , 1. ]])

prog_005000

Is there any predefined camera values that we need to use in case we generated smpl from ROMP? Please do let me know

ALso, if theres a way to generate accurate smpl for HUMAN3.6m do let me know, since its a multi-camera video

louhz commented 1 year ago

My suggestion is that you can follow the dataset setting in A-nerf, which provide convention between zju and human3.6 m

Since in zju-mocap, the RH and Th are extracted from the 72 poses info, and in classic SMPL like ROMP, it is placed in the first three digit of 72 poses. In the case of camera poses, you should check the coordinate of camera, in humannerf case, you need to transfer every camera poses in to opencv coordinate.

Also for Kp3d, I remember this is actually the bone coordinate to world coordinate transformation matrix I think the Instant-avator or A-nerf can provide more details.

Dipankar1997161 commented 10 months ago

Sorted it out @louhz. Error was the camera values as the ray projection were inaccurate due to it. Thanks again