[Bug Report] scale_sparsity_penalty_by_decoder_norm option is ignored

jbloomAus / SAELens

Training Sparse Autoencoders on Language Models

https://jbloomaus.github.io/SAELens/

MIT License

481 stars 127 forks source link

[Bug Report] scale_sparsity_penalty_by_decoder_norm option is ignored #363

Closed chanind closed 1 week ago

chanind commented 2 weeks ago

Describe the bug The scale_sparsity_penalty_by_decoder_norm config option is never read. Sparsity is always scaled by the decoder norm regardless of what value is set for this parameter.

Thanks to @demiantill for finding this!