norabelrose
commented
1 month ago I assume the transfomer layer index start from 1, and embedding layer index is 0.
This is not true. The layers are zero-indexed. We simply use get_submodule to select the appropriate layer.
https://huggingface.co/EleutherAI/sae-llama-3-8b-32x-v2
from sae import Sae sae = Sae.load_from_hub("EleutherAI/sae-llama-3-8b-32x-v2", hookpoint="layers.24")
What is the "layers.24" mean here? the 24th or 25th transfomer hidden states output? I assume the transfomer layer index start from 1, and embedding layer index is 0.