AlignmentResearch / tuned-lens

Tools for understanding how transformer predictions are built layer-by-layer
https://tuned-lens.readthedocs.io/en/latest/
MIT License
438 stars 47 forks source link

Support for code coverage #40

Closed levmckinney closed 1 year ago

levmckinney commented 1 year ago

Adds support for code coverage to the package. Ideally this should enforce 100% coverage on tests and increasing coverage for all subsequent pull requests.

norabelrose commented 1 year ago

Is this like a ratcheting thing where it never lets the coverage go down in a new PR? I agree we need more coverage than we currently do although I remember codecov was sort of a pain in the ass when contributing to imitation

levmckinney commented 1 year ago

Is this like a ratcheting thing where it never lets the coverage go down in a new PR? I agree we need more coverage than we currently do although I remember codecov was sort of a pain in the ass when contributing to imitation

At the moment all it does is make sure that overall code coverage does not go down. I'm inclined to configure it to require X% code coverage for all new lines added by a pull requests. I'm thinking like 95%, so you don't have to worry about covering every error, but most paths are getting covered.

codecov[bot] commented 1 year ago

Codecov Report

:exclamation: No coverage uploaded for pull request base (main@af55452). Click here to learn what that means. The diff coverage is n/a.

:exclamation: Current head 1c07299 differs from pull request most recent head 0816a2f. Consider uploading reports for the commit 0816a2f to get more accurate results

@@           Coverage Diff           @@
##             main      #40   +/-   ##
=======================================
  Coverage        ?   41.06%           
=======================================
  Files           ?       34           
  Lines           ?     2077           
  Branches        ?        0           
=======================================
  Hits            ?      853           
  Misses          ?     1224           
  Partials        ?        0           

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more