Closed osanseviero closed 2 weeks ago
@chanind would be curious for your thoughts on this / whether this is something you might have capacity to do :)
This looks like a cool idea! I guess we just need to open a PR like specified in that example to add sae-lens to the list of libraries? Seems doable!
I'm worried about how to specify the snippet though - it looks like we know the model ID, but to load a SAE you need to specify the sae_id, not just the model_id, which I don't think the snippet would have a way of knowing. Also we rename models for use with the SAE.from_pretrained()
function. I think we'd need to make it so that if the user passes in a huggingface repo_id to SAE.from_pretrained()
, we fallback to pulling from that huggingface repo? Then the snippet could look something like the following:
from sae_lens import SAE
sae = SAE.from_pretrained("blah/my_sae", <fill in sae ID>)[0]
thoughts?
Yes, it's just opening a PR as the example. Yes, I think that would work well :+1:
Maybe with these small changes?
from sae_lens import SAE
sae = SAE.from_pretrained("blah/my_sae", sae_id=<fill in sae ID>)
Hey all! 👋 Now that v3.14.0 is out, which means Gemma Scope works with .from_pretrained
, it would be great to extend this to any SAELens model on the Hugging Face Hub.
This would entail:
config.yaml
model_id
instead of release
.sae.save_pretrained()
or sae.save_config()
could be good.and would mean that:
Easier + auto-generated code snippets on the HF hub, like:
.
We will add this to huggingface.js as a follow-up to a previous PR (https://github.com/huggingface/huggingface.js/pull/826) which added SAELens as a supported library. This will also ensure download counts are correctly tracked for the model, like:
Let me know what you think @jbloomAus!
I'm excited about this. @chanind will follow up with you to discuss.
Proposal
Hey there! It would be great to have better support for SAELens on the HF Hub and provide
The repo to add this is open source (huggingface.js). See here an example PR to add a new library
Checklist