Open dynamicwebpaige opened 2 years ago
This would be so cool, I really like the user story you made with the side-by-side benchmark!
We were talking about integrating something like this but into tasks pages, so the workflow would be:
And maybe when you select a particular task on /models we could add a link to the task page:
This is probably the simplest way of doing it but I understand that it's not the same as having it directly integrated into the /models page.
So maybe we want to go further and do it exactly like you said: integrate it directly into the /models page: you drag/input a picture/audio/text and all the visible models on the page compute and it switches to "benchmark mode" (that could be gamechanger 🤯). That will of course be a lot of work and I'm not even sure that we can hold that many computations at the same time 👀 (edit: we will find a way 👍 ).
🗒️ Motivation
When a user selects a specific task on the Hugging Face Hub - for example,
image-to-text
:That user is shown a series of models, with no guidance as to which model might be state of the art, or which might be the most performant for their use case.
To test the capabilities and behavior of each model, the user must:
Space
or a Colab notebook available (not every model does).🙏 Desired Behavior
The user should be able to:
image-to-text
).image-to-text
).