search

facebookresearch / pytorch3d

PyTorch3D is FAIR's library of reusable components for deep learning with 3D data
https://pytorch3d.org/
Other
8.85k stars 1.32k forks source link

How is the zbuffer gradient computed? Why is it non-zero if it's just pulled from OpenGL? #1906

Open MartensCedric opened 1 month ago

MartensCedric commented 1 month ago

Unexpected behavior

Looking at DIB-R: https://research.nvidia.com/labs/toronto-ai/DIB-R/files/diff_shader.pdf, which this library seems to have a similar implementation, when calculating the derivative of pixel color with respect to vertex position, the derivatives of barycentric coordinates are computed to get smooth gradients (for foreground pixels). This library appears to be doing something similar, but does not do this for the depth calculation.

In the case that I'm optimizing only a depth map (I don't care about texture, and shading), I saw that the HardDepthShader just takes the zbuffer from OpenGL (essentially from the shader). It does not compute the depth from the barycentric coordinates of a face (it could because the depth of each vertex can be known from the camera parameters).

This means that the gradient of the image color with respect to vertex position seems incorrect, or at least computed differently. I expected the derivative dDepth/dVertexPos to always be 0. However, I have to admit that I am not getting this in my code, which is unexpected. I would like to know whether or not the depthmap gradient is computed like I understand.

bottler commented 1 month ago

Can you share some code? How do you initialize the rasterizer and shader?