ApolloResearch / rib

Library for methods related to the Local Interaction Basis (LIB)
MIT License
3 stars 0 forks source link

Simplify test_ablation setup #291

Closed danbraunai-apollo closed 6 months ago

danbraunai-apollo commented 6 months ago

Currently, our tests in test_ablations require the following process: For both mnist and modadd:

It will be better if we instead:

  1. Run the Rib build on the fly before computing the ablations. Note that the ablations only run on a couple of layers, so we can get away with using fewer layers in the rib build and also probably smaller n_intervals.
  2. If the above is still too slow, have a script which runs them, changes the tlens_model_path and mlp_path to a relative path, and saves them in sample_graphs.

It seems like we can do option 1 for the tests very easily, and even keep it as a fast-running test. But the downside is that the default ablation configs use the sample_graphs, so we probably want these to be up to date anyway.

Best path might be to implement 1 but also add a script which trains the models and stores them in sample_graphs. Unsure when we should run this script.

nix-apollo commented 6 months ago

I agree it would be nice to have at least some test ablation scripts run rib build before ablations. When doing so it would be great for some of them to use a centered rib build (this would increase test coverage in ablations.py)