how to balance different loss?

[xingmimfl]

hello, it's really a great work! and I'm trying to re-implement this paper . During my training, it seems very hard to balance different loss. How did your choose weights to the loss? let different loss to be the same value or make one loss larger while another smaller?

[YuDeng]

Hi. My strategy for weight balancing is quite simple. First, I train a model with some initial loss weights (for example all equals to 1). After that, I adjust the weights for different loss parts based on their numerical values in the first training step so that different loss parts could have similar value (ratio between 0.1~10) after multiplying their corresponding weights. Then, I train new models based on the adjusted weights. Finally, I might do some minor adjustments to get a better performance.

[xingmimfl]

thanks a lot! I will try it :P

[xingmimfl]

Hi. My strategy for weight balancing is quite simple. First, I train a model with some initial loss weights (for example all equals to 1). After that, I adjust the weights for different loss parts based on their numerical values in the first training step so that different loss parts could have similar value (ratio between 0.1~10) after multiplying their corresponding weights. Then, I train new models based on the adjusted weights. Finally, I might do some minor adjustments to get a better performance.

sorry to disturb you again. where do you put parameters and texture regularization term? in the loss? I know that some times parameters regularization was put in the optimizers.

[YuDeng]

I also add them into the total loss.

[xingmimfl]

I also add them into the total loss.

thanks. :P