search

AlliedToasters / dfencoder

BSD 3-Clause "New" or "Revised" License
129 stars 49 forks source link

How do you handle your losses different scales #4

Open EtienneT opened 4 years ago

EtienneT commented 4 years ago

Hi,

Thanks you for the very interesting library! Looking at the code I was wondering how you were balancing the different losses scales. The MSE loss is probably on a completely different scale than the cross-entropy loss. How do you make sure that the MSE loss does not dominate compared to your other losses?

I never actually implemented this myself, but I found two papers that implement custom loss weights. This enables you to have any number of losses each with different scales, but then each gets its own weight for the calculation of the total loss.

The paper in question are section 3 of Multi-Task Learning Using Uncertainty to Weigh Losses and then section 2 of Auxiliary Tasks in Multi-task Learning where some other authors refine the formula a bit more from the first paper.

AlliedToasters commented 3 years ago

Sorry for the late reply!

Currently, there is no logic within the library for weighting or re-scaling the loss for different output values - this is a really good point and an important feature to add! So far, I've found the implementation as-is works pretty well for purposes of feature extraction - it can be improved, though!