search

coqui-ai / STT-models

Open models for Coqui STT
https://coqui.ai
117 stars 35 forks source link

WER should be more easily accessible #2

Open dabinat opened 3 years ago

dabinat commented 3 years ago

Some models have >85% WER and others have <15% WER but it is hard to know which models are reasonably usable without clicking on each one individually. It would be helpful to have WER as a sortable column on the Models page of the site.

Sometimes WER information is hard to find even on the model's card page. Some models direct you to other pages to view that information and the ones that do display it show it in different formats. It would be good if there was some form of standardization on how to report WER within the model cards.

kdavis-coqui commented 3 years ago

@dabinat Do you know if there's some standard way to add metadata to releases?

We use tags to hold some of that info (e.g. the language, creator, and model version) but it's a bit of a hack. It there were a standard way to add metadata we could then add WER and other info, then allow these to be sorted on in the website.

dabinat commented 3 years ago

I don’t know exactly how you have things set up server-side but git tags can be annotated and I guess you could make that annotation message a JSON object or other parseable format.

Or each release could contain an info.json file that the server then caches.

kdavis-coqui commented 3 years ago

To generate the web interface to the models we use the GitHub REST API, in particular the "get a release"[1] results. As far as I can tell, tag annotations are (unfortunately) not returned from such a "get release call".

Having each release contain a info.json is definitely possible; however, it's slightly awkward. But, it's looking more and more like that's the only solution.