Differentiable Renderer Tutorial

[armindocachada]

Hello, First of all, I wanted to say, what amazing work you are all doing in 3D Deep Learning. The DIB-R and GanVerse3D papers are amazing Also, many thanks for this wonderful Deep Learning library, which I am eager to learn.

I followed the new "Differentiable renderer" tutorial and I was able to run it successfully. So no issues there. However, I have some doubts/questions about this tutorial.

1. I was able to find the training data generated by Omniverse for the clock, however, I can't find the original 3D model that was used to generate this training data. It would have been nice to include it in Github so we can also repeat the steps in the tutorial needed to generate the training data.

2. I am a bit confused about the goal of the tutorial itself. My previous understanding about DIB-R is that you can train it with 3D data or annotated data like the one coming from ShapeNet, Pascal3D and Bird CUB dataset and then you can do inferencing on images it hasn't seen before. In this way, if I trained DIB-R with a dataset about cars, I should be able to reconstruct the 3D model from a 2D photo after I train DIB-R. But this tutorial is not showing us how to do that? Could you please clarify how would this tutorial fit-in if I was trying to use DIB-R to learn on my own dataset?

[Caenorst]

Hi @armindocachada ! Thank you for your interest in Kaolin and your kind words.

1. The models can be found here: https://graphics.pixar.com/usd/downloads.html, we usually avoid distribute data because most of them are under license, however, we can add the link above somewhere in the documentation or tutorial :+1:
2. I completely understand the confusion, as I had it myself sometimes. There are two things to separate:
   1. DIB-R the differentiable renderer
   2. The applications featured in DIB-R publication

Here we are focusing on the differentiable renderer, showing a very simple use case, which is shape and texture optimization.

One of the reason is that the Kitchen dataset is that the Kitchen set is a lighter dataset so it's easier for you to try the tutorial.

But we do understand the appeal for applications showed in DIB-R (which I would agree are amazing :rocket: ) and we will add them on the Model Zoo which is a work-in-progress, we don't have a clear ETA, but we are trying to go as fast as possible and will keep you posted.

[armindocachada]

Thanks for your response. Now it makes more sense, and I know where to download the model from. I am still trying to get my head around DIB-R and your clarification definitely helps.

Also, looking forward to see the models from the paper in the model zoo.

Another question, is the dataset-gan that was used for GanVerse3d going to be available anytime soon in the Koallin library? Or is it going to be part of the model zoo?

[armindocachada]

Hello,

I tried the tutorial again but replaced the clock with a bowl from the kitchen set. Seems like DIB-R had a bit of trouble reconstructing the 3D model. Any idea what I am missing?

[Caenorst]

Concerning the full DIB-R with GANverse3d, we will put that into the model-zoo repository. We are still not sure about ETA, and we are not 100% sure we can legally provide pretrained models however we will provide full script. Given the hype we are bumping it in our priority list :)

For the bowl, this is not surprising given how naive the example is, it will struggle with concave shape. My guess is that we may force roughness to 1 to improve performance. We can also start to randomize light position, and use the light position as input (apply light with https://github.com/NVIDIAGameWorks/kaolin/blob/f82436433eb7e07f7dfdbb79f38ee9926a76bedc/kaolin/render/mesh/utils.py#L63-L108). The DIB-R training in the model zoo will be way more resistant to that :)

[armindocachada]

Thanks for your patience in answering my questions. I think I have finally understood what DIB-R is and what it is not.