mozilla / translations

The code, training pipeline, and models that power Firefox Translations
https://mozilla.github.io/translations/
Mozilla Public License 2.0
154 stars 33 forks source link

Display corpus size in W&B #529

Closed eu9ene closed 1 month ago

eu9ene commented 6 months ago

We should display things we look at often in W&B. Final merged corpus size after deduplication is something I look at periodically to understand how aggressive the cleaning is overall. We can also display corpus size after each cleaning stage as we discussed with @gregtatum which should probably be a part of the analysis job.

La0 commented 4 months ago

As discussed today, we would expose either

eu9ene commented 4 months ago

We could also provide the final OpusTrainer config to the parser. It includes paths to the training datasets.

eu9ene commented 4 months ago

I think the idea of this ticket was also to display the size of the corpus after different cleaning steps but we can start with uploading only the size of the final corpus.