search

zengwang430521 / DecoMR

Repository for the paper " 3D Human Mesh Regression with Dense Correspondence "
Apache License 2.0
168 stars 23 forks source link

Theory behind ' Ensure the neighboring pixels of vt on the UV map are meaningful' #13

Closed wangzheallen closed 3 years ago

wangzheallen commented 3 years ago

Hi Wang, I am trying to understand the bary-weight calculation process. Basically you are trying to mapping the uv space faces to xyz space faces and want to solve the weight between them. However, I start to get lost from https://github.com/zengwang430521/DecoMR/blob/master/models/uv_generator.py#L179 and fail to understand why and what is the theory/intuition behind that about the calculation of w, u, v?

I know from https://github.com/zengwang430521/DecoMR/blob/master/models/uv_generator.py#L205 you are trying to construct a mask in UV space, but the lines before that seems confusing to me.

zengwang430521 commented 3 years ago

Sorry, line 173-210 is a little dirty. I did not explain it clearly.

These codes are used to alleviate the problem in the resampling of vertices. Let me describe the problem first:

When resample the vertex location from a UV map, bilinear interpolation is utilized. This may cause a problem on the vertices near the edges of UV parts.

As shown in the figure below, the red and green vertex are interpolated using the 4 grid pixels. However, 3 of them will have zero value because there are outside of any triangle, which will cause artifacts on the resampled mesh.

grid_tri

In the codes of densbody-pytorch, they dilate the UV map for a pixel when generating the UV map. However, we found that dilation may cause some new artifacts when the UV map resolution is not high enough. It may mix multiple UV parts.

In addition, our uv_generator assumes every pixel to be related with only 3 vertices. So we are not able to use the pre-calculated dilation weight. And we can only realize the dilation process after the location map is generated, which is space-consuming.

So we use another way to solve this problem. When generating the location map, for any grid pixel that is used in the resampling process, if it has zero value, we assign it with the value of the vertex near to this pixel. For example: the 4 grid pixels are used in the resampling process of the red vertex, but 3 of them are with zero value. So these 3 pixels should be assigned with the value of the red vertex.

Some pixels may be related with multiple vertices. For example: the 4 grid pixels are related with both red and green vertex. In this condition, we decide the pixel value according to the interpolation weight.

wangzheallen commented 3 years ago

Thanks for the kind clarification! This solves my concern! Will the operation of 'replacing the grid value with nearest vertex' results in non-smooth/erratic mesh? as suggested by https://gitee.com/seuvcl/CVPR2020-OOH/issues/I1PALL?from=project-issue

zengwang430521 commented 3 years ago

In our experiments, this may leads to a few erratic vertices in low resolution UV map. If high resolution is used, such as 256×256, both dilation and our process is okay.

wangzheallen commented 3 years ago

Thanks for the answer!