Closed AndresCasado closed 4 years ago
I added thickness to the mesh so all the visible faces have the same orientation, but it didn't solve the problem. So I don't think it's related to the face orientation.
Using the mesh with thickness plus backface culling keeps showing differences between renderings too.
(Top row is differentiable rendering in both images)
After checking the zbuffer of the first column I learned that the nearest fragments had positive values, while the farthest fragments had negative values.
Then I read the blending function, and the z-buffer related calculations seemed strange to me. If I need more weight for the front faces, why does the function give more weight to the back ones?
So I did an experiment, what if I reverse the function so it gives more weight to the front fragments? This is my code:
z_inv = (fragments.zbuf - zfar) / (znear - zfar) * mask
z_inv_max = torch.max(z_inv, dim=-1).values[..., None]
weights_num = prob_map * torch.exp((z_inv - z_inv_max) / blend_params.gamma)
znear
and zfar
are initialized to the max and min values of the zbuffer
This is the result:
Now the first and last columns are the ones that have the correct result!
This reinforces my assumption that the problem is z-buffer related.
I'm going to check if the z-buffer of all the points of view have the same values (high in front, low in back), because I think a possible cause is that the renderer is using world or object z-depth instead of camera z-depth.
After checking the zbuffers I confirmed the problem is that they were in world coordinates instead of view coordinates. This issue was supposedly fixed (see #210 ) in commit f0ba4c5, but said commit does not belong to any branch, and the file currently in master
branch does not look like the code modified in that commit ( https://github.com/facebookresearch/pytorch3d/blob/master/pytorch3d/renderer/mesh/renderer.py )
Strangely enough, my local install has a file similar to the renderer.py
modified in f0ba4c5. After creating my own version of the MeshRenderer
with that fix, and reverting the changes I made to the blending function, the differentiable image looks just like it should.
What steps should be taken now? Should I rename this issue? Close it and reopen #210?
@AndresCasado thanks for pointing this out. The barycentric clipping and zbuf interpolation has now been moved to the rasterization cuda kernel. There is a setting called clip_barycentric_coordinates
in the RasterizationSettings
. Have you set this to True
? The default is False
.
Previously in https://github.com/facebookresearch/pytorch3d/commit/f0ba4c53e58f20ab13db5f4f3a283b5f03b8d8ed we had set barycentric clipping to happen automatically whenever the blur_radius
was greater than 0 but now it has to be set manually in the RasterizationSettings
.
Let me know if this resolves your issue!
Thanks for the explanation. Now I understand why the code is different. I had an old version installed. I thought I updated it to make sure it wasn't an already solved issue but it seems I messed up somewhere, sorry.
I've just reinstalled PyTorch from master and it works just fine, so issue solved. Thanks again, @nikhilaravi.
Hi. After learning how to correctly use the differentiable render (after my previous issue #236 ) I've been able to get some results for my project, but now I've hit another wall, as it seems meshes with holes are problematic.
Example (code and .obj below) Rendering a sphere with some holes returns different shaded results. The top row are rendered with a differentiable render, while the bottom row are rendered with a normal renderer.
Click to see full image![Imgur](https://i.imgur.com/wK54XkC.jpg)
It looks to me like there is a problem with the z-depth of the fragments, but I can't pinpoint where. I've searched the issues but I haven't seen an exact solution. I think it could be related to #116 #149.
I've tried some things:
These things are what make me think this has something to do with the z-depth of the fragments, although it's strange that the 6th column is correct.
Before noticing this problem with holes I already had some suspicions about my settings, because the shading of my differentiable images was strange (first and last columns would still be different even without holes). So this may be a symptom of something else.
Reproduce
I've uploaded the .obj file to https://pastebin.com/G0s47z0V