search

ApolloResearch / rib

Library for methods related to the Local Interaction Basis (LIB)
MIT License
3 stars 0 forks source link

Train, ribify and analyse modadd with layernorm #221

Closed danbraunai-apollo closed 9 months ago

danbraunai-apollo commented 9 months ago

We used to have an lm_rib_build config for modular arithmetic with layer norm, but it used some outdated config vals (float32, truncation_threshold=1e-5, and the model was trained with an old transformerlens that used IGNORE=-1e5).

To reproduce with the latest updates, the following needs to be done:

  1. Train a new model with the standard config as is checked in but with normalization_type = "LNPre"
  2. Run lm_rib_build on the standard config, changing the tlens_model_path to the path of the layer normed model.

I (Dan) still think it's quite important to try and understand a simple, overparameterised model trained with layer norm.

danbraunai-apollo commented 9 months ago

Copied over to #118. Closing here