Question regarding the backpropagation and differentiability

Hi all,

I'm using PyTorch3D to get the silhouette from a 3D human model so that I'm able to optimize the human model parameters (e.g. poses and global translation) over a loss term on the predicted silhouette and the ground truth silhouette.

The issue is that the gradient of the loss w.r.t. the optimized parameters (thetas and trans) is equal to zero all the time. I checked the computation graph and the gradient of the loss w.r.t. the silhouettes is non-zero but after that, the gradient will stay zero. I guess this is caused by either the mapping from mask_mesh to mask or from smpl_output.vertices to mask_mesh because the mapping from the parameters to smpl_output is already proven to be differentiable.

Could someone give a solution to this problem? Thanks a lot in advance!

Here is the code snippet (just the relevant part):

class SMPLBlendPar(NamedTuple):
    sigma: float = sys.float_info.epsilon # 0
    gamma: float = 1e-4
    background_color: Sequence = (1.0, 1.0, 1.0)

class SMPLRenderer(nn.Module):
    def __init__(self, batch_size, image_size, f=50, R=None, t=None, model_path=None, gender='male'):

        self.image_size = image_size
        self.f = f
        self.smpl_mesh = SMPL(model_path=model_path, batch_size=batch_size, gender=gender)
        self.cam_R=R
        self.cam_T=t
        self.faces = torch.from_numpy(self.smpl_mesh.faces[None, :, :].astype(int)).cuda().int()

        self.blend_params = SMPLBlendPar()
        self.device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
        torch.cuda.set_device(self.device)

        self.cameras = SfMPerspectiveCameras(focal_length=self.f, R=self.cam_R, T=self.cam_T, device=self.device)
        self.raster_settings = RasterizationSettings(image_size=image_size,faces_per_pixel=10,blur_radius=np.log(1. / 1e-4 - 1.) * self.blend_params.sigma)

        self.rasterizer = MeshRasterizer(cameras=self.cameras, raster_settings=self.raster_settings)

        self.mask_shader = SoftSilhouetteShader(blend_params=self.blend_params)

        self.mask_renderer = MeshRenderer(rasterizer=self.rasterizer, shader=self.mask_shader)

    def forward(self, betas, thetas, trans, displacement=None, mode='a'):

        smpl_output = self.smpl_mesh.forward(betas=betas, body_pose=thetas[:,3:],
                                             transl=trans,
                                             global_orient=thetas[:,:3],
                                             return_verts=True)        

        verts_rgb = torch.ones_like(smpl_output.vertices)

        mask_texture = Textures(verts_rgb=verts_rgb.cuda())

        mask_mesh = Meshes(verts=smpl_output.vertices,faces=self.faces.expand(b,-1,-1),textures=mask_texture).cuda()

        mask = self.mask_renderer(mask_mesh)[...,3] 

        return mask

The loss is just the naive squared error: mask_loss = torch.sum((silhouettes - gt_silhouettes)**2)

facebookresearch / pytorch3d

Question regarding the backpropagation and differentiability #344