neelnanda-io
commented
1 year ago I'm happy to put the weights on HuggingFace if you think Redwood would be fine with it?
On Fri, 24 Feb 2023, 12:33 ArthurConmy, @.***> wrote:
Since Redwood's 2L model has been open https://github.com/redwoodresearch/rust_circuit_public-sourced https://github.com/redwoodresearch/remix_public, anyone should be able to download weights: https://drive.google.com/file/d/1CEZF9QEY2VEtOatgVh7VRL4SkDYyIFLL/view?usp=sharing
and use them in this library:
%% [markdown]
import transformer_lens import os from transformer_lens.HookedTransformerConfig import HookedTransformerConfig from transformer_lens.HookedTransformer import HookedTransformer
cfg = HookedTransformerConfig( n_layers=2, d_model=256, n_ctx=2048, n_heads=8, d_head=32,
model_name : str = "custom"
# d_mlp: Optional[int] = None # act_fn: Optional[str] = None d_vocab=50259, # eps: float = 1e-5 use_attn_result = True, use_attn_scale = True, # divide by sqrt(d_head) # use_local_attn: bool = False # original_architecture: Optional[str] = None # from_checkpoint: bool = False # checkpoint_index: Optional[int] = None # checkpoint_label_type: Optional[str] = None # checkpoint_value: Optional[int] = None # tokenizer_name: Optional[str] = None # window_size: Optional[int] = None # attn_types: Optional[List] = None # init_mode: str = "gpt2" # normalization_type: Optional[str] = "LN" # device: Optional[str] = None # attention_dir: str = "causal" attn_only = True, # seed: Optional[int] = None # initializer_range: float = -1.0 # init_weights: bool = True # scale_attn_by_inverse_layer_idx: bool = False positional_embedding_type = "shortformer", # final_rms: bool = False # d_vocab_out: int = -1 # parallel_attn_mlp: bool = False # rotary_dim: Optional[int] = None # n_params: Optional[int] = None # use_hook_tokens: bool = False)
et_model.load_state_dict(torch.load("et_model_state_dict.pt"))
would be great if someone could check that this works, then maybe someone at @redwoodresearch https://github.com/redwoodresearch should add the model weights to HF or something?
— Reply to this email directly, view it on GitHub https://github.com/neelnanda-io/TransformerLens/issues/180, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASRPNKLS5UTXMIAQJM6OP33WZCTDJANCNFSM6AAAAAAVG3DCSE . You are receiving this because you are subscribed to this thread.Message ID: @.***>
daniel-ziegler
ArthurConmy
Since Redwood's 2L model has been open-sourced, anyone should be able to download weights: https://drive.google.com/file/d/1CEZF9QEY2VEtOatgVh7VRL4SkDYyIFLL/view?usp=sharing
and use them in this library:
would be great if someone could check that this works, then maybe someone at @redwoodresearch should add the model weights to HF or something?