[BUG] `FisherMCLinearOperator` is off by a factor of `C` when using `MSELoss` or `BCEWithLogitsLoss` with mean reduction

f-dangel / curvlinops

PyTorch linear operators for curvature matrices (Hessian, Fisher/GGN, KFAC, ...)

https://curvlinops.readthedocs.io/en/latest/

MIT License

18 stars 8 forks source link

[BUG] `FisherMCLinearOperator` is off by a factor of `C` when using `MSELoss` or `BCEWithLogitsLoss` with mean reduction #111

Closed runame closed 6 months ago

runame commented 6 months ago

We scale the loss by 1/C here, but also implicitly here for MSELoss and here for BCEWithLogitsLoss. This was not caught by tests since the expensive tests in test_fisher.py are skipped for the GitHub workflow.