According to the Arxiv paper https://arxiv.org/pdf/1609.07093.pdf, the orthogonal regularization should be:
orth_loss = orth_loss + (reg * sym.abs().sum())
instead of:
orth_loss = orth_loss + (reg * sym.sum())
since absolute opration was performed as per the paper.
without abs, the loss will introduce negative value as well.
According to the Arxiv paper https://arxiv.org/pdf/1609.07093.pdf, the orthogonal regularization should be:
orth_loss = orth_loss + (reg * sym.abs().sum())
instead of:orth_loss = orth_loss + (reg * sym.sum())
since absolute opration was performed as per the paper. without abs, the loss will introduce negative value as well.