search

mosaicml / llm-foundry

LLM training code for Databricks foundation models
https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm
Apache License 2.0
3.83k stars 502 forks source link

Error if metadata matches existing keys #1313

Closed dakinggg closed 2 days ago

dakinggg commented 6 days ago

We flatten the metadata so that it is easier to parse out of mlflow/run metadata. Because of this, a user's metadata key could overwrite the real key in the config that gets logged. This PR fixes that by simple erroring and asking the user to rename their metadata key. It also adds flatten_metadata to allow not flattening the metadata.

Original run that overwrites the key: metadata-with-error-4-G9ps3T Run with duplicate key that now errors: metadata-unflat-1-ltIBGL Run without flattening metadata: metadata-unflat-2-VoxpxS

josejg commented 5 days ago

Could we prefix metadata_ in the metadata keys for clarity ?

josejg commented 4 days ago

looks good to me, but I'll give it a try just in case