NVlabs / nvdiffrast

Nvdiffrast - Modular Primitives for High-Performance Differentiable Rendering
Other
1.31k stars 139 forks source link

Acquiring vertex depth #57

Closed jby1993 closed 2 years ago

jby1993 commented 2 years ago
Thank you for releasing this powerful tool!
Is there a robust way to get visible vertex or faces? I found that the face_ids acquired from the rasterizer can be sparse for dense meshes, I guess this is because some triangle is too small to overlap one pixel. 
So how to get the complete visible triangles and vertex? I think one way is to utilize the depth buffer, but how to get the depth of mesh vertices?
s-laine commented 2 years ago

This was touched in issue #55 recently. The short answer is that there is no robust way to determine all faces or vertices that would be visible to the camera using rasterization. Increasing resolution decreases the chance of missing a vertex, but it never goes to zero.

To determine fully accurate vertex visibility, you would need to test geometrically that a line segment between a vertex and the camera does not intersect any scene geometry. Determining geometrically accurate face visibility is considerably more difficult, as a face can be visible even if all its vertices are hidden.

To get the depth buffer value of a scene vertex you need to compute z/w of the clip-space coordinate you feed to rasterize() in the pos argument. For example, depth = pos[..., 2] / pos[..., 3] gives you an array with per-vertex depth values that are in the same space as the z/w component of rasterizer output.

I think it's an interesting idea to use depth buffer to include missed vertices from dense meshes. I believe this should work nicely as long as the surface isn't too curved. If you have the time, please let me know afterwards if you got it working.