This PR adds documentation to the HBP extension of Conv2d layers, which is responsible to compute KFAC/KFLR/KFRA. The docstrings draw connections to the notation in the KFC paper, and outline important differences, as well as improvements for consistency. It also adds a test case for KFAC, KFLR for which both approximations become exact.
Note to myself: I made notes how to connect Hessian backpropagation to KFAC for convolutions by imposing a Kronecker structure on the backpropagated quantity. This concept can also be applied to KFRA to achieve more consistency, but is currently not done by the code.
This PR adds documentation to the
HBPextension ofConv2dlayers, which is responsible to computeKFAC/KFLR/KFRA. The docstrings draw connections to the notation in the KFC paper, and outline important differences, as well as improvements for consistency. It also adds a test case forKFAC, KFLRfor which both approximations become exact.Note to myself: I made notes how to connect Hessian backpropagation to
KFACfor convolutions by imposing a Kronecker structure on the backpropagated quantity. This concept can also be applied toKFRAto achieve more consistency, but is currently not done by the code.