search

zengwang430521 / DecoMR

Repository for the paper " 3D Human Mesh Regression with Dense Correspondence "
Apache License 2.0
168 stars 23 forks source link

Data pre-processing about generating UV map #8

Closed wangzheallen closed 3 years ago

wangzheallen commented 3 years ago

May I ask what is the physical meaning behind line 45-46, is that orthographic projection? or transformation to UV space? https://github.com/zengwang430521/DecoMR/blob/master/datasets/preprocess/generate_gt_iuv.py#L45

and https://github.com/zengwang430521/DecoMR/blob/master/datasets/preprocess/generate_gt_iuv.py#L52 why do you have '1 - uv_tmp[:, :, 0]'

Additionally, I agree it is continuous among different semantic body parts in your UV map, which makes the learning easier. Is the frontal view and back view of the same finger continuous in your defined UV map? as in Fig 3 in original paper.

zengwang430521 commented 3 years ago

Hi,

Line 45-46 aim to normalize the 2D joints coordinates to [-1, 1]. And then we can calculate the camera parameters according to the normalized 2D joints by line 48 cam, err = cal_cam(origin_2d, target_2d) . With the camera parameters, we can then render the IUV image using the GT mesh. In addition, cal_cam is NOT a perfect algorithm to get the camera parameters, we just get an approximative result.

The reason for 1 - uv_tmp[:, :, 0] is a little complex. In one word, this is caused by the property of the renderer we used:

Given a mesh and UV coordinates of every texture vertex (denoted as U_vert and V_vert), we want to produce a 2-channel UV image, where the background area is filled with value 0 and the foreground area is filled with the value UV_vert.

However, in the rendered UV image: In U channel, background area is filled with value 1, and foreground area is filled with value=U_vert. In V channel, background area is filled with value 0, and foreground area is filled with value=1 - V_vert.

This is not what we want. So we let UV_vert = 1 - UV_vert in the line 70 of preprocess_datasets.py and then let uv_im[:, :, 0] = 1 - uv_tmp[:, :, 0] in line 52. Then we can get the right UV image.

Actually, the frontal view and back view of the same finger is NOT continuous in our UV map. Such discontinuity is inevitable in 2D UV map. We have tried to use spherical mapping but found that spherical mapping tends to loss details of the mesh. Maybe more advance spherical mapping algorithm can solve this problem.

wangzheallen commented 3 years ago

Thanks for the clarification, this resolves my concern :-)

wangzheallen commented 3 years ago

Is the purpose of multiply 255 to generate human-readable image? https://github.com/zengwang430521/DecoMR/blob/master/datasets/preprocess/generate_gt_iuv.py#L57 If so, when do you normalize back to [-1,1]? I did not find it here: https://github.com/zengwang430521/DecoMR/blob/master/datasets/base_dataset.py#L209

zengwang430521 commented 3 years ago

The generated UV image is in scale [0, 1]. We multiply the UV value with 255, so we can save the IUV image as a png file. Saving the IUV image as npy file costs too much storage space.

We normalize the UV value back to [0, 1] in the training process. https://github.com/zengwang430521/DecoMR/blob/f96526ba9547d23cddd5931fa3fe75a49927e3f3/train/trainer.py#L104 https://github.com/zengwang430521/DecoMR/blob/f96526ba9547d23cddd5931fa3fe75a49927e3f3/train/trainer.py#L172

wangzheallen commented 3 years ago

Thanks for the clarification!