search

mosaicml / llm-foundry

LLM training code for Databricks foundation models
https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm
Apache License 2.0
3.99k stars 525 forks source link

Loss v len callback #1226

Closed ShashankMosaicML closed 4 months ago

ShashankMosaicML commented 4 months ago

Adds the loss v len callback that logs loss vs context length for each iteration into tables. Using those tables we can plot graphs such as the following:

Screenshot 2024-05-22 at 2 58 54 PM