no gradient when using the depth output only in rasterization

[iszihan]

Hello,

I am trying to build something that computes losses using the depth channel output by the rasterize() function but this seems to not generate gradients. Am wondering if this is expected or is there a way to get gradient from this output channel alone?

Thank you in advance!

[s-laine]

The z/w output channel of the rasterizer indeed doesn't propagate gradients backwards. This is intentional and mentioned in the documentation. The reason is that these values are not very useful except for depth comparisons between surfaces, which is what they are used for in the antialiasing op.

The simple way to determine world- or camera-space position, or camera-space z, for each pixel, is to add the relevant values as vertex attributes in the interpolation phase. This way they will propagate gradients as expected.

[iszihan]

That makes sense, thank you for the answer!

[nishadgothoskar]

I am trying to do pose estimation using an inverse graphics approach on depth data. So in this case it would be very useful to have gradients over the depth channel. Have there been attempts to do this?

[s-laine]

The depth channel of the rasterizer contains the post-projective (z/w) values that are not directly useful for geometry optimization. They are nonlinear, they depend on camera near/far clip plane positions, etc., so you'd need to manually invert the effects of the perspective transformation before you get anything useful out of them.

Instead, I recommend extracting the per-vertex camera-space depth values after all model/view transformations, but without the perspective transformation. This per-vertex depth can then be interpolated using the interpolate() op, and the gradients will propagate correctly.

A reply here appears to contain an example of interpolating a depth value, but I haven't tested if it works.