Tatoeba data source should index by lemmas

Mindful / cardbuilder

Making flashcard construction effortless and vocabulary acquisition (slightly) less painful 🎴

https://pypi.org/project/cardbuilder/

MIT License

11 stars 1 forks source link

Open Mindful opened 3 years ago

Mindful commented 3 years ago

Tatoeba data source doesn't currently lemmatize tokens when it indexes them for example sentence search, but it should.

This can be seen in the _split_sentence function set in the __init__ method of TatoebaExampleSentences