Improve language profile generation

komodojp / tinyld

Simple and Performant Language detection library for NodeJS

https://komodojp.github.io/tinyld/

MIT License

415 stars 12 forks source link

Improve language profile generation #3

Closed kefniark closed 3 years ago

kefniark commented 3 years ago

Description

To be more accurate, an algorithm like this one needs big datasets and processing abilities. Currently the script execution time is getting longer and longer but only on one core.

Multi threading and multi processing could allow us to ingest more data and get a better detection accuracy

kefniark commented 3 years ago

with 1.1.0 most language data are cached and not generated every time.

This allow really fast profile generation and iteration