Perhaps consider replace `geometric_median_dataset` with `nn.BatchNorm1d(x, affine=False)`

ai-safety-foundation / sparse_autoencoder

Sparse Autoencoder for Mechanistic Interpretability

https://ai-safety-foundation.github.io/sparse_autoencoder/

MIT License

171 stars 39 forks source link

Perhaps consider replace `geometric_median_dataset` with `nn.BatchNorm1d(x, affine=False)` #179

Open wassname opened 8 months ago

wassname commented 8 months ago

Perhaps consider replacing geometric_median_dataset with nn.BatchNorm1d(x, affine=False) layer It doe a similar thing with less parameters and complexity.