Model Ladder

[dirkgr]

Adds a script that should let us establish scaling properties of interventions we're considering.

TODO:

• [x] Final eval doesn't get created
• [x] Run all sizes and fix what breaks.
• [ ] Add a 3B config?
• [x] See if using torch.compile() is trustworthy.
• [x] See if we can optimize perf in other ways.
• [ ] The 300M, 750M, and 1B all have 16 heads and 16 layers. Shouldn’t those vary in scale a little?
• [x] Make it work on only one node
• [x] Make it possible to continue runs (i.e., add a --load_path flag)
• [x] Pick a perplexity set to target
• [ ] Build a perplexity set that's based on the downstream tasks we care about
• [ ] Find out why the downstream tasks are so random. Logits almost the same?
• [ ] Find out if there are differences between 1B with FSDP and 1B with DDP