Is giving loss on z_buffer able to flow?

[Water-Silver]

Hi! Thanks for sharing great work. I've been using it to reconstruct facial models. I have one question though. Is giving constraints on z_buffer (e.g. mse(z_buffer, depth)) able to back propagate till the shape coefficients?

I have altered the return value of rasterize_clip_space() in rasterizer.py to return z_buffer. But somehow, even though I give a mse loss for the z_buffer and the depth map, the loss does not seem to flow.

And I realize variables like z_buffer_grads in your code, but also found ctx.mark_non_differentiable(triangle_ids, z_buffer) in RasterizeTriangle.py.

How should I change the code to make the loss given on z_buffer flow?

Sorry for short understanding on your code. It would greatful if you could give me some tips.

[shty32]

Hi, I'm sorry for the late reply.

Indeed, this code does not support a differentiable z_buffer currently. As you mentioned, we force the z_buffer non-differentiable by ctx.mark_non_differentiable(triangle_ids, z_buffer) and the variable "grad_z_buffer" is a placeholder, since our code and the original TF code did not implement this part.

However, I have another idea that may help you. Since our code can generate triangle_ids on each pixel, you may use this id to trace the corresponding triangle and use barycentric to compute the depth for this pixel based on xyz of this triangle.

Based on the above idea, here is a temporary way, i.e., you may use xyz (may need to be normalized) to fill the vertex color and use ambient light only (set to 1), then the three channels of the rendered image may represent x y z coors.

Please let me know if it works or you have further questions.

[Water-Silver]

Thanks for replying back!

I think your idea would work. I've found an implementation of rendering depth image but in tensorflow, (in case you wonder https://github.com/jiaxiangshang/MGCNet) and their implementation and your idea seem to match.

Thanks for your help!