search

pmh47 / dirt

DIRT: a fast differentiable renderer for TensorFlow
MIT License
312 stars 63 forks source link

Differentiability wrt vertex colors #104

Open justanhduc opened 3 years ago

justanhduc commented 3 years ago

Hi @pmh47. Thanks for sharing this wonderful project. I wonder whether Dirt is differentiable wrt vertex colors? I tried to learn some uv but I get None when trying to get the gradients. Thanks in advance!

pmh47 commented 3 years ago

Yes, it is. If you define your model something like in those in the samples folder, it should be differentiable wrt vertex colors, or UVs and texture values. If you have a short code example where it fails, then paste it here and I can take a look.

justanhduc commented 3 years ago

Thanks. I found the problem. I have this code fragment 

self.vertex_colors = tf.Variable(np.random.rand(6890, 3).astype('float32'), trainable=True)
vertex_colors = self.vertex_colors
renderer_color = RenderLayer(self.img_size, self.img_size, 3, vertex_colors, np.ones(3), self.faces,
                             [self.img_size, self.img_size], [self.img_size / 2., self.img_size / 2.],
                             name='render_color_layer')
self.rendered_color = [renderer_color(tf.stop_gradient(v)) for i, v in enumerate(self.vertices)]

in which the RenderLayer is Dirt. The gradient works fine for self.vertex_colors and my loss decreases. However, if I apply some transformation on self.vertex_colors, no matter how trivial it is, like vertex_colors = self.vertex_colors + 1, the loss won't change at all and self.vertex_colors doesn't change. Could you please let me know what potential problem there can be?

pmh47 commented 3 years ago

There is nothing wrong with the code you pasted -- I can't see why the gradients would be zero here. Hence there may be a problem with your RenderLayer class itself, or elsewhere in your code. If you can provide a minimal runnable example that prints the incorrect gradient (and tell me which version of tensorflow & python you use), I can look at what's happening, but I suspect it's not a problem with DIRT itself.