adds scripts to prepare training/evaluation data for a language identification model and then train the langid model
updates lang identifier class to use the new (v3) floret/fasttext model rather than the existing (v2) thinc/cld3
adds additional datasets to training data, for more and better language coverage
updates a couple existing datasets to newer versions
adds floret (the explosion folks' wrapper around fasttext) as a package dependency
Motivation and Context
I wanted to update my home-brewed language identification code to use something a bit more standard and, ideally, faster / more accurate. I also want to use floret for other purposes, so this use case brings it into textacy for later.
How Has This Been Tested?
Screenshots (if appropriate):
Types of changes
[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to change)
Checklist:
[ ] My code follows the code style of this project.
[ ] My change requires a change to the documentation, and I have updated it accordingly.
Description
floret(the explosion folks' wrapper aroundfasttext) as a package dependencyMotivation and Context
I wanted to update my home-brewed language identification code to use something a bit more standard and, ideally, faster / more accurate. I also want to use
floretfor other purposes, so this use case brings it intotextacyfor later.How Has This Been Tested?
Screenshots (if appropriate):
Types of changes
Checklist: