search

NVlabs / nvdiffrast

Nvdiffrast - Modular Primitives for High-Performance Differentiable Rendering
Other
1.35k stars 144 forks source link

Camera pose optimization supported? #5

Closed yxie20 closed 3 years ago

yxie20 commented 3 years ago

Hello all! Thank you again for open-sourcing this powerful tool.

I am wondering that camera pose optimization is trivially supported by nvdiffrast? The rendering pipeline would look like this:

I went ahead and implemented a training loop, but the training outcome wasn't always successful. Camera either sometimes drifts away or simply doesn't converge. Other times the camera can find the optimal solution. Generally, the camera has a higher chance of drifting away when the initial error is large.

My guess is that anti-aliasing gradients (as opposed to soft rasterization), can only reason at a very local, pixel-level scale and cannot reason at a far distance away from an object. Hence the need for gradient-free greedy optimization in samples/pose.py?

Thank you!

s-laine commented 3 years ago

Hi @yxie20 - for geometric optimization (e.g., vertex coordinates or camera position) you should include an antialiasing step at the end of the rendering pipeline. Otherwise you won't obtain gradients related to triangle visibility/coverage, as the coverage is discontinuous w.r.t. vertex positions without antialiasing. The bit about antialiasing in the documentation gives more details, and Section 3.6 in the paper describes the actual implementation in case you're curious.

satyajitghana commented 2 years ago

@yxie20 hey, did it work ? I've tried pytorch3d for the same, but failed. my loss becomes NaN and the camera drifts away after some time. I'm thinking of trying out this.