tfjgeorge / nngeometry

{KFAC,EKFAC,Diagonal,Implicit} Fisher Matrices and finite width NTKs in PyTorch
https://nngeometry.readthedocs.io
MIT License
203 stars 20 forks source link

layer norm work in progress #80

Closed tfjgeorge closed 7 months ago

codecov[bot] commented 7 months ago

Codecov Report

Attention: 4 lines in your changes are missing coverage. Please review.

Comparison is base (5c61d0f) 95.47% compared to head (1d61aed) 95.23%.

Files Patch % Lines
nngeometry/layercollection.py 75.00% 4 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #80 +/- ## ========================================== - Coverage 95.47% 95.23% -0.24% ========================================== Files 31 31 Lines 3579 3655 +76 ========================================== + Hits 3417 3481 +64 - Misses 162 174 +12 ``` | [Flag](https://app.codecov.io/gh/tfjgeorge/nngeometry/pull/80/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Thomas+George) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/tfjgeorge/nngeometry/pull/80/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Thomas+George) | `95.23% <95.00%> (-0.24%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Thomas+George#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

tfjgeorge commented 7 months ago

this should add LayerNorm as discussed in https://github.com/tfjgeorge/nngeometry/issues/79

Tests are implemented for tiny fully connected and conv architectures, but for some reasons implementing LayerNorm broke the unit tests for custom layers (Cosine and WeightNorm). I will continue investigating