Implement Hoyer-Square along with sqrt L^0.5 regularization

DavidUdell / sparse_circuit_discovery

Circuit discovery in GPT-2 small, using sparse autoencoding

MIT License

6 stars 1 forks source link

Closed DavidUdell closed 8 months ago

DavidUdell commented 9 months ago

The sparse autoencoder world rolls on. Reportedly, these along with my existing loss terms improve autoencoder quality. H/t @tkwa

DavidUdell commented 8 months ago

Deprioritized for now.