Can I do backward using Pytorch3d?

[cha-noong]

❓ Questions on how to use PyTorch3D

When I Training a 3d generation model, the result of this model is obj (tensor), and Using pytorch3d, I want to obtain the rendering image. so I want to calculate the loss(between render image <-> GT) and apply backward to the 3d own's model

Previously, I wanted to use the numpy-based rendering library, but the backward calculation was difficult so it could not be used.

pytorch3d look like based on tensor, so it seems easier. Is it possible?

[nikhilaravi]

Hi @Chanwooong this is exactly the use case for which we built PyTorch3D. We support gradients for almost all operators, and for differentiable rendering, we have custom CUDA kernels for fast and efficient calculation of the forward and backward pass.

You can call .backward() directly after you calculate the loss as you described e.g.

# generate mesh from 3D generation model
# generate image using renderer e.g. image = renderer(mesh)
loss = torch.sum((image - image_gt) ** 2)
loss.backward()

To get started try one of the tutorials e.g. camera position optimization which explains how to use the renderer in an optimization loop.