Wrap `FileNotFound` exceptions in the finetuning dataloader and `convert_text_to_mds`

mosaicml / llm-foundry

LLM training code for Databricks foundation models

https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm

Apache License 2.0

3.84k stars 503 forks source link

Wrap `FileNotFound` exceptions in the finetuning dataloader and `convert_text_to_mds` #1246

Open angel-ruiz7 opened 1 month ago

angel-ruiz7 commented 1 month ago

Creates two exceptions:

DatasetMissingFileError --> Tells the user that a dataset file could not be found during a failure in the finetuning dataloader where split / path were not defined properly
DatasetInvalidFolderError --> Tells the user that an input_folder is invalid and while trying to list objects in convert_text_to_mds.py

mapi exceptions implemented in https://github.com/databricks-mosaic/mcloud/pull/4088

irenedea commented 1 month ago

@jjanezhang @angel-ruiz7 This is awesome! I was actually just seeing a lot of FileNotFound errors last week. Are there any plans to handle FileNotFound errors that are surfaced when downloading checkpoints? I saw some errors last week due to a bad custom weights path.

(To be clear, this is out of scope for this PR, just wanted to get your thoughts 😄)