search

wznpub / MMA_Regularization

official code for paper "MMA Regularization: Decorrelating Weights of Neural Networks by Maximizing the Minimal Angles"
MIT License
12 stars 2 forks source link

mathematical question about Eq(7) #2

Open ChineseYjh opened 1 year ago

ChineseYjh commented 1 year ago

Hello~ sorry for bothering you!

How is the sin derived from the norm in Eq(7)? Moreover, the gradient (LHS of Eq(7)) should be a vector, but the result seems to be a scalar (RHS of Eq(7)).

Looking forward for your reply~

wznpub commented 1 year ago

hello， thanks for your interest @ChineseYjh

Equation(7) is about the Euclidean norm of the gradient， the norm should be a scalar rather than a vector.

Note that M_wi represents the projection matrix of w_i. For detail, pls refer the Equation (11) of paper "PR Product: A Substitute for Inner Product in Neural Networks". https://openaccess.thecvf.com/content_ICCV_2019/papers/Wang_PR_Product_A_Substitute_for_Inner_Product_in_Neural_Networks_ICCV_2019_paper.pdf

Thank you very much

ChineseYjh commented 1 year ago

hello， thanks for your interest @ChineseYjh

Equation(7) is about the Euclidean norm of the gradient， the norm should be a scalar rather than a vector.

Note that M_wi represents the projection matrix of w_i. For detail, pls refer the Equation (11) of paper "PR Product: A Substitute for Inner Product in Neural Networks". https://openaccess.thecvf.com/content_ICCV_2019/papers/Wang_PR_Product_A_Substitute_for_Inner_Product_in_Neural_Networks_ICCV_2019_paper.pdf

Thank you very much

Thank you very much~