search

nateraw / modelcards

📝 Utility to create, edit, and publish model cards on the Hugging Face Hub. [**Now lives in huggingface_hub**]
MIT License
15 stars 4 forks source link

Handle invalid model-index instead of throwing data away #48

Open nateraw opened 2 years ago

nateraw commented 2 years ago

Right now, if model-index is not formatted correctly in the remote repo when using ModelCard.load(repo_id), it will tell you the model-index is invalid and that we're not loading eval results.

For context, under the hood modelcards.CardData uses a list of EvalResult to represent the model index, which is stored at CardData.eval_results. When you turn card data to dict, it doesn't return the list of EvalResult, but instead formats it to a valid model index and assigns it to the model-index key of the metadata dict.

The code can be seen here:

https://github.com/nateraw/modelcards/blob/a45b85bab516531813b0f80db4399a56de4447b9/modelcards/card_data.py#L151-L167

So, maybe in the case that the found 'model-index' is invalid, we assign it to card_data['model-index'] so it can still be accessed. If the user tries to update the eval_results, we'll use that, but if not, the previous model-index will still be in tact.

nateraw commented 2 years ago

Perhaps we can still create the eval_results if formatted somewhat correctly (throwing a warning that its invalid). This way it could make it easier for a user to fix invalid model-index in their existing cards.