search

f-dangel / curvlinops

PyTorch linear operators for curvature matrices (Hessian, Fisher/GGN, KFAC, ...)
https://curvlinops.readthedocs.io/en/latest/
MIT License
18 stars 8 forks source link

Feature: Exponential moving average for `KFACLinearOperator` #151

Open f-dangel opened 1 month ago

f-dangel commented 1 month ago

It would be good to add a method exponential_moving_average that takes two KFACLinearOperators and produces another KFAC whose Kronecker matrices were combined using an exponential moving average. This 'smoothing' scheme is extremely common in practise.