search

jaeill / CVPR23-VNE

Official Implementation of CVPR 2023 paper: "VNE: An Effective Method for Improving Deep Representation by Manipulating Eigenvalue Distribution"
MIT License
24 stars 3 forks source link

Confusion pertains to the autocorrelation matrix #2

Closed Brownchen closed 10 months ago

Brownchen commented 10 months ago

I did not find any calculations related to the autocorrelation matrix in the released code. Moreover, it appears that the VNE regularizer is calculated using the eigenvalues of the output matrix of the projectors. Additionally, I am seeking clarification on how you implemented the observations of eigenvalue distribution throughout the entire paper.

It is possible that my understanding is flawed, and I would greatly appreciate it if you could help clarify my thoughts.

Thanks!

jaeill commented 10 months ago

Regarding the autocorrelation matrix, kindly direct your attention to the PyTorch implementation in Section C (Main Algorithm) of our paper. The rho denotes the autocorrelation matrix. The symbol rho is designated in accordance with the notation convention of the density matrix in quantum information theory, as specified in Equation (9). Notably, two distinct implementations of the get_vne function are presented. It is easy to verify that these implementations are mathematically equivalent, yet the second variant demonstrates superior computational efficiency when the batch size, denoted by N, is less than the representation dimension, denoted by d.

Regarding the distribution of eigenvalues, kindly examine the distribution of eig_val, particularly in its ordered form, in the get_vne function.

Brownchen commented 10 months ago

Thanks for your kind response!