jbloomAus / SAELens

Training Sparse Autoencoders on Language Models
https://jbloomaus.github.io/SAELens/
MIT License
481 stars 127 forks source link

fix: hotfix scale decoder norm is not passed to training sae #377

Closed chanind closed 1 week ago

chanind commented 1 week ago

Description

In #365, we fixed a bug where the scale_sparsity_penalty_by_decoder_norm was being ignored and the SAE was always scaling by decoder norm regardless. However, this fix revealed a second bug where we're not passing the scale_sparsity_penalty_by_decoder_norm param through to the training SAE at all. This sort of bug is easy to happen given that we create the TrainingSAEConfig from the runner config via creating a dictionary without type checking.

This PR adds a test just that scale_sparsity_penalty_by_decoder_norm is now being passed through correctly to get this fix out asap, but I'll make a follow-up PR with a more robust fix in the form of better tests or type checking or something after this is merged.

Type of change

Please delete options that are not relevant.

Checklist:

You have tested formatting, typing and unit tests (acceptance tests not currently in use)

codecov[bot] commented 1 week ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 72.74%. Comparing base (17506ac) to head (4f28b21). Report is 2 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #377 +/- ## ======================================= Coverage 72.74% 72.74% ======================================= Files 22 22 Lines 3266 3266 Branches 431 431 ======================================= Hits 2376 2376 Misses 762 762 Partials 128 128 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.