Questions about loss_norm

[aoyanl]

Hi, I wonder what is the effect of the loss_norm? According to the training log ,the loss value of each item is 1 when self.loss_norm is True. Could you provide me some explanation about the motivation and the effect ? Thanks! https://github.com/JeffWang987/OpenOccupancy/blob/2c0dc831fa2899315c227a78bf33d00cc3f4ce80/projects/occ_plugin/occupancy/detectors/occnet.py#L225

[JeffWang987]

Hi, I wonder what is the effect of the loss_norm? According to the training log ,the loss value of each item is 1 when self.loss_norm is True. Could you provide me some explanation about the motivation and the effect ? Thanks!

https://github.com/JeffWang987/OpenOccupancy/blob/2c0dc831fa2899315c227a78bf33d00cc3f4ce80/projects/occ_plugin/occupancy/detectors/occnet.py#L225

The loss_norm is a typical strategy to avoid tunning the loss weights, it is motivated by the multi-task learning community. You can turn it off, which may slightly decrease the performance. Or you can carefully design the loss weights by yourself, which may enhance the performance (but we did not experiment with it).

[qihao-plus]

@JeffWang987 If the loss_norm is set True, the loss is always 1. How to check the loss curve change ?

[JeffWang987]

@JeffWang987 If the loss_norm is set True, the loss is always 1. How to check the loss curve change ?

For clarifying:

1. Firstly we can train with the original loss: L = L1 + L2 + L3, it is observed that the pretrained model can generate reasonable/satisfactory performance.
2. To enhance the performance (and avoid tuning loss weights), we can train with loss_norm: L = L1 / L1.detach() + L2 / L2.detach() + L3 / L3.detach(). Note that each loss item is 1.
3. For checking the loss curve, one can adjust the loss_norm code by setting: L = L1 + L2 / L2.detach() L1.detach() + L3 / L3.detach() L1.detach(). Then you can observe loss changing based on the first loss item.

[qihao-plus]

Thanks!