Literature review on scaling laws

[RaymondLi0]

Get an idea of the different flavours of scaling-law works that are out there. Any work that tries to estimate the optimal scale of model and dataset size, with regards to a certain metric (PPL, or other) Some references:

• Kaplan et al: Scaling Laws for Neural Language Models
• Henighan et al: Scaling Laws for Autoregressive Generative Modeling
• Hernandez et al: Scaling Laws for Transfer
• Tay et al: Scale efficiently: Insights from pre-training and fine-tuning transformers
• Tay et al: Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?
• Tay et al: Transcending Scaling Laws with 0.1% Extra Compute
• Hoffmann et al: Training Compute-Optimal Large Language Models
• Bansal et al: Data Scaling Laws in NMT: The Effect of Noise and Architecture
• Clark et al: Unified Scaling Laws for Routed Language Models

[RaymondLi0]

Particularly, it would be good to know if some previous work examined downstream performance instead of PPL in a scaling-law setting

[keyboardAnt]

Additional papers:

• Bavarian et al: Efficient Training of Language Models to Fill in the Middle (OpenAI, Jul 2022)

[RaymondLi0]

Other related work:

• Wei et al: https://arxiv.org/pdf/2206.07682.pdf. Although they do not do "scaling laws", they study how capabilities of LLMs are affected by scale