pmh47
commented
3 years ago In short, yes, regularisation (or careful parameterisation) is important to avoid nasty self-intersecting meshes. In [1], we use local smoothness regularisers, and also show a parameterisation that avoids self-intersections a priori. We can avoid a silhouette loss even on natural images with texture by restricting the space of 'allowed' shapes for the foreground to live in some low-dimensional (learnt) manifold, using a VAE setup. Note that these techniques belong to the model rather than the renderer (DIRT) itself. Hence, I think that the same techniques will work with renderers other than DIRT (at least, the ones that have reasonably well-behaved derivatives everywhere) -- it does not have any particularly unusual properties.
The only reference for how DIRT works is a small section of my PhD thesis [2].
[1] Leveraging 2D data to learn textured 3D mesh generation, Henderson et al. CVPR 2020, https://arxiv.org/abs/2004.04180 [2] Section 3.4 of https://era.ed.ac.uk/handle/1842/35600
Hi, I notice that in your paper it is emphasized that silhouette loss is not needed (which is a really nice feature!). However, I also notice that all most all of the recent differentiable mesh renderer requires silhouette loss along with other bunch of regularizer to prevent the generation of self-intersecting faces.
I am curious about why this is the case. Can dirt at the same time optimize mesh shape and mesh vertex color (texture with non-uniform color) without silhouette loss (in the paper only mesh shape optimization is shown)? Is there anywhere that I can find a more detailed introduction of the technique that you employed to build dirt?
Thanks!