search

ai-safety-foundation / sparse_autoencoder

Sparse Autoencoder for Mechanistic Interpretability
https://ai-safety-foundation.github.io/sparse_autoencoder/
MIT License
191 stars 39 forks source link

Add multiprocessing support to datasets #150

Closed alan-cooney closed 11 months ago

alan-cooney commented 11 months ago

Useful where the data is sharded across files, as multiple files can be processed in parallel (when pre-downloaded).