Add `fisher_type="forward-only"` (FOOF method) option to `KFACLinearOperator`

f-dangel / curvlinops

scipy linear operators for the Hessian, Fisher/GGN, and more in PyTorch

https://curvlinops.readthedocs.io/en/latest/

MIT License

17 stars 5 forks source link

Add `fisher_type="forward-only"` (FOOF method) option to `KFACLinearOperator` #98

Closed runame closed 6 months ago

runame commented 6 months ago

Add a fisher_type to implement the preconditioner used for FOOF. This represents an extreme value for this argument since we only do a forward pass, compute the input covariances, and set the gradient covariances to identity matrices.

coveralls commented 6 months ago

Pull Request Test Coverage Report for Build 8450734252

Details

7 of 8 (87.5%) changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage decreased (-0.02%) to 89.491%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
curvlinops/kfac.py	7	8	87.5%
<!--	Total:	7	8	87.5%	-->

Totals
Change from base Build 8395557809:	-0.02%
Covered Lines:	1090
Relevant Lines:	1218

💛 - Coveralls

runame commented 6 months ago

I like the proposed test and have just added it. Please check the math in the docstring.