komodojp / tinyld

Simple and Performant Language detection library for NodeJS

https://komodojp.github.io/tinyld/

MIT License

415 stars 12 forks source link

Improve for long text/document #2

Closed kefniark closed 3 years ago

kefniark commented 3 years ago

Description

Split long text into smaller chunks (by punctuation .,;:()
Only process a max number of chunk
Merge result before picking best guess

kefniark commented 3 years ago

with 1.1.0:

text are automatically split in multiple chunk
for long document, only few chunks are evaluated, not the whole text
results are merged