search

NVlabs / nvdiffrast

Nvdiffrast - Modular Primitives for High-Performance Differentiable Rendering
Other
1.35k stars 144 forks source link

silhouette loss possibility #14

Closed esx2ve closed 3 years ago

esx2ve commented 3 years ago

Hello and thank you for releasing this amazing library!

I'm experimenting with implementations of different approaches to joint geometry optimization and inverse rendering, and I've tried implementing a sillhouette (mask) loss approach. Geometry optimization with texture works very well, but when I input a ones tensor into the texturing step and compare against the alpha mask of the input (synthetic data, so I have ground truth mask), no gradient seems to flow into the vertices (or anywhere). I also tried differentiable gaussian blur and it didn't seem to help. Is it in principle possible, or am I completely off road trying this?

Thank you!

s-laine commented 3 years ago

Hi @esx2ve! Are you executing the antialiasing op before computing the loss? If not, there will be no coverage-related gradients flowing into vertex positions because the coverage is point sampled and discontinuous. The antialiasing step creates a continuous relationship between vertex positions and colors of silhouette pixels, enabling gradient flow. If you see no gradients despite using the antialiasing op, please send a simple test case that reproduces the problem.

esx2ve commented 3 years ago

Hello @s-laine, thank you for the promt reply!

I've located the problem - I've been masking the resulting image after the antialiasing. Once I've moved it before the antialiasing step, everything works!