search

EleutherAI / sae-auto-interp

https://blog.eleuther.ai/autointerp/
Apache License 2.0
97 stars 11 forks source link

[Improvement] - Add functionality to the feature cache saving function. #2

Closed SrGonao closed 4 months ago

SrGonao commented 4 months ago

We want to be able to save stuff locally (we can keep the upload stuff if you think it is useful). I think we want to be able to save all the features (saving just x features is ok for debugging I guess, and for proof of concept stuff), so I'm torn on saving features individually.

I was saving a big tensor for all the activations and a big tensor for all the locations. What do you think about still supporting that? I'm pretty sure that saving 130k+ small files per layer is going to be a nightmare.

I think it would be cool to be able to add more activations to a pile that we already have, but that maybe should be a different issue?

cadentj commented 4 months ago

Resolved. See save_splits function under cache. It will accept a "n_splits" argument and saves chunks of features per layer.

Also included a save_selected_splits which accepts a tensor of features and saves those features to a tensor.

SrGonao commented 4 months ago

Yup