search

facebookresearch / vicreg

VICReg official code base
MIT License
516 stars 87 forks source link

DxD instead of NxN in the computation of the covariance matrix of Z? #25

Open aarontyliu opened 1 year ago

aarontyliu commented 1 year ago

Hi, I came across the implementation of the covariance matrix of Z while trying to use VICReg in my project. I feel like I am missing something here, but I wonder if the implementation should be x @ x.T instead of x.T @ x?

Thanks for any information you can provide.

steinate commented 10 months ago

Same quetions. Did you figure out? I also think the former implementation is right.

hfangcat commented 10 months ago

The implementation should be x.T @ x, if x's dimension is (B, D), here B: batch_size, D: dimension, because they want the covariance matrix to be a function of the dimension