Fixing sequence_id =-1 bug, adding tests

mosaicml / llm-foundry

LLM training code for Databricks foundation models

https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm

Apache License 2.0

3.83k stars 502 forks source link

Fixing sequence_id =-1 bug, adding tests #1324

Closed ShashankMosaicML closed 2 days ago

ShashankMosaicML commented 3 days ago

If there is padding in the input sequences, the LossPerpVLen metric throws an error. This PR fixes it. Thanks @dakinggg for reporting this and helping debug.