search

thuml / LogME

Code release for "LogME: Practical Assessment of Pre-trained Models for Transfer Learning" (ICML 2021) and Ranking and Tuning Pre-trained Models: A New Paradigm for Exploiting Model Hubs (JMLR 2022)
MIT License
202 stars 18 forks source link

A bug in the implemention of fixed point algorithm #23

Open haohq19 opened 1 year ago

haohq19 commented 1 year ago

Hi, I'm working on your paper and find a bug in _fit_fixed_point in LogME.py. One part of the calculation of evidence in line 146 is:

- 0.5 * np.sum(np.log(alpha + beta * sigma)) \

which is to calculate $-0.5\log{|A|}$ by multiply all the eigenvalues. The variable sigma here have the shape of (k, 1), where k is the rank of the feature matrix. However, matrix A have the shape of (D, D) so sigma here should have the shape of (D, 1), with the last D - k elements equal to zero.

This bug will cause a wrong result when N < D, in which case k<=N<D and the calculation of det(A) will be wrong.

Hope this might be helpful :)

youkaichao commented 1 year ago

Oh, thank you for your careful reading! Indeed the N < D case is less considered. I think the bug can be fixed if we pad sigma to have the shape of D, so that the result can be fixed after broadcasting. What do you think?

haohq19 commented 1 year ago

Yes, I agree with your solution :)

youkaichao commented 1 year ago

Great, would you mind open a pull request for that?

haohq19 commented 1 year ago

Yes I'd love to. I've pulled a request for the solution. Thank you very much.