AlignmentResearch / tuned-lens

Tools for understanding how transformer predictions are built layer-by-layer
https://tuned-lens.readthedocs.io/en/latest/
MIT License
432 stars 47 forks source link

Added support for rank as prediction trajectory metric #105

Closed levmckinney closed 1 year ago

levmckinney commented 1 year ago

This PR adds support to for rank as a metric that can be visualized using a prediction trajectory. Rank in this case is the rank of the target token relative to all the other tokens in the predictions at each layer in the prediction trajectory. This can be accessed using the new rank method. In addition, this PR adds support for a log scaling the metrics when visualizing a specific TrajectoryStatistic.

codecov[bot] commented 1 year ago

Codecov Report

Merging #105 (2feccf7) into main (c9921f5) will increase coverage by 0.06%. The diff coverage is 82.60%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #105      +/-   ##
==========================================
+ Coverage   80.88%   80.95%   +0.06%     
==========================================
  Files          32       32              
  Lines        2077     2116      +39     
==========================================
+ Hits         1680     1713      +33     
- Misses        397      403       +6     
Impacted Files Coverage Δ
tuned_lens/plotting/trajectory_plotting.py 84.94% <42.85%> (-3.43%) :arrow_down:
tuned_lens/plotting/prediction_trajectory.py 91.87% <77.77%> (-0.48%) :arrow_down:
tests/plotting/test_prediction_trajectory.py 100.00% <100.00%> (ø)