kevinzakka / pytorch-goodies

PyTorch Boilerplate For Research
603 stars 71 forks source link

Orthogonal regularization is wrong #7

Closed huangzhii closed 4 years ago

huangzhii commented 4 years ago

According to the Arxiv paper https://arxiv.org/pdf/1609.07093.pdf, the orthogonal regularization should be: orth_loss = orth_loss + (reg * sym.abs().sum()) instead of: orth_loss = orth_loss + (reg * sym.sum()) since absolute opration was performed as per the paper. without abs, the loss will introduce negative value as well.

kevinzakka commented 4 years ago

Good catch, thanks @huangzhii !