Normalisation of the key-point residual loss for integration with PVNET

[Divyam10]

Hello @BoChenYS and @jonbakerfish ,

First of all, Thanks for publishing code for your insightful paper.

I am trying to train pvnet along with the loss function defined in you demoPoseEst.py. However, the loss is way too high in value compared to the other two losses in PVNET. I tried to normalise it by dividing it by Σ(y - y_mean)² where y is the batch of ground truth key_points and y_mean is the mean Keypoint. Normalized RSS = RSS / Σ(y - y_mean)²

However, the value is still too high in the range of 1 to 100. Meanwhile other losses are less than 0.01 generally.

How do I solve this? Or is there any other way I can incorporate BPNP in Clean-Pvnet.

Thanks!

[BoChenYS]

You can always use different coefficients for different loss terms, to balance their scale and impact on the learning process. Does that make sense?

[Divyam10]

Okay, makes sense! A constant coefficient would do? Or is there an adaptive method to go about it?

[BoChenYS]

Constant coefficients will do.

[Divyam10]

Thanks for the advice and swift response. @BoChenYS I tried training the Clean-Pvnet model with a the loss multiplied by the constant coefficients. However, after few steps the Bpnp loss increases from a magnitude in the range of 1 to 10 to 20k-40k. I have used the co-efficient to be 0.0001.
Due to this the overall loss results in either NaN or INF. I tried different values of the coefficients but the same this persists. the gradient either exploded or vanishes.

# Key - point Voting Loss
        weight=batch['mask'][:, None].float()
        vote_loss=self.vote_crit(output['vertex'] * weight, batch['vertex'] * weight, reduction='sum')
        vote_loss = vote_loss / weight.sum() / batch['vertex'].size(1)
        loss += vote_loss

# Segmentation Loss
        mask = batch['mask'].long()
        seg_loss = self.seg_crit(output['seg'], mask)
        loss += seg_loss

        scalar_stats.update({'loss': loss})
        image_stats = {}

        kpt_2d_pd = output['kpt_2d'] 
        kpt_2d_gt = batch['kpt_2d']

# BPNP - Loss 
        P_out = self.bpnp(kpt_2d_pd, self.Kpt_3D, self.K)
        pts2d_pro = BPnP.batch_project(P_out, self.Kpt_3D, self.K)
        loss_bpnp = ((pts2d_pro - kpt_2d_gt)**2).mean()  + ((pts2d_pro - kpt_2d_pd)**2).mean()
        loss += loss_bpnp* 0.0001

I have added the function which calculates all the three losses. Any insights or conceptual errors would really help!

Thank you