search

quickwit-oss / whichlang

A blazingly fast and lightweight language detection library for Rust
MIT License
353 stars 13 forks source link

Is it about faster and slightly more accurate than lingua-rs? #9

Open LeMoussel opened 1 year ago

LeMoussel commented 1 year ago

Does Whichlang is about faster and slightly more accurate than lingua-rs?

Rem: it should be noted that lingua-rss supports 75 languages compared to the 16 languages supported by whichlang

getreu commented 1 year ago

whatlang currently supports 68 languages.

pemistahl commented 1 year ago

Hi, I'm the author of lingua-rs. I've just released a new version of my library together with a detailed accuracy evaluation and performance benchmark.

  1. Yes, Whichlang needs much less processing time than Lingua. My benchmark yields the following results for determining the language of 2,000 sentences in Whichlang's 16 supported languages. Lingua is restricted to the same set of 16 languages.
Single Thread Multiple Threads
Lingua / high accuracy mode 333.31 ms 37.347 ms
Lingua / low accuracy mode 180.54 ms 24.550 ms
Whichlang 2.0458 ms 351.03 µs


  1. No, Whichlang is not more accurate than Lingua. On the contrary, Lingua still is the most accurate language detection library for Rust.
Average Detection Performance

All benchmark results are available here and here.

LeMoussel commented 1 year ago

@pemistahl How did you limit Lingua to the same set of 16 languages?

getreu commented 1 year ago

LanguageDetectorBuilder in lingua - Rust