Open peterbe opened 7 years ago
By the way, my hack is here: https://github.com/peterbe/langdetect Careful, I haven't done go for a long time.
I noticed this while I was testing and I haven't found a solution yet. I added it to the TODO list in this commit.
For what it's worth, I tested something similar with a different project (https://pypi.python.org/pypi/guess_language-spirit) and it's based on a bunch of trigrams too. I can't remember the example but it suffered the same problem.
I just checked it out and tested it with the same phrases you used.
>>> guess_language("Le candidat socialiste à l’élection présidentielle.")
'fr'
Correct!
>>> guess_language("Mitt namn på svenska är Peter")
'sv'
Correct!
>>> guess_language("testing in english")
'UNKNOWN'
Not correct.
>>> guess_language("wondering still if it works in english")
'af'
Still not correct.
Considering the trigrams used in that project are not similar to the ones I used ... wtf is wrong with English ?
I tried franc too.
var franc = require('franc');
console.log(franc("Le candidat socialiste à l’élection présidentielle."))
console.log(franc("Mitt namn på svenska är Peter"))
console.log(franc("testing in english"))
console.log(franc("wondering still if it works in english"))
output;
▶ node test.js
fra
swe
uig
nld
Short inputs like "happy" "hello" also returns incorrect results.
Looks like the longer the string, the better the result?
Hilarious:
func TestLangDetection(t *testing.T) {
lang := whatlanggo.DetectLang("english english english english english english english")
if lang != whatlanggo.Eng {
t.Fatalf("Expected lang to be %v but was %v", whatlanggo.LangToString(whatlanggo.Eng), whatlanggo.LangToString(lang))
}
}
--- FAIL: TestLangDetection (0.00s)
search_test.go:12: Expected lang to be eng but was uzb
FAIL
Providing a Whitelist via DetectLangWithOptions
helps.
Algorithm:
Quote from the article:
Disadvantages:
May provide falsy results for short texts (smaller than 200-300 letters). Whatlang tries to compensate this with is_reliable attribute.
So there are 2 ways to fix/mitigate this:
First, thanks for making this library available, it helped me to get rid of a C dependency from cld2.
Just wanted to add a few additional data points concerning non-detection of English text:
Hi. I am a creator of the original library in Rust. I just want to say, that you should not have high expectation, if input text is relatively small. The library is based on statistical profiles of languages (trigrams). The bigger input text, the better it represents its statistical profile.
There is nothing you can do with this issue.
Some possible solutions however:
@miku
Just wanted to add a few additional data points concerning non-detection of English text:
Rust version at the moment provides is_reliable
boolean in result. When it's true
it is guaranteed, that language is recognized correctly. Otherwise you should not trust the result. For all of your text samples it returns result is_reliable=false
.
Doesn't fix the problems with detection, but now Info{} has the confidence rating.
I took your awesome lib and wrapped it in a little command line app. I also added a conversion table from ISO 639-3 to ISO 639-1.
Correct!
Correct!
But....
Not right.
Not right either.
Also, would it be possible to output a list of probabilities? That way my app, where I hope to use this, could throw warnings if the probabilities "aren't certain enough".