We're using tinyld to detect the language on some external content, which generally works great. I'm looking to move to the heavy version to fix a detection error, but I noticed that one of our test cases started to fail. It's based on the title of https://www.scienceopen.com/hosted-document?doi=10.14293/S2199-1006.1.SOR-.PPL3VEC.v2, which contains a couple of spelling mistakes.
The normal version correctly identifies it as Spanish:
But the heavy version seems to decide it's Galician pretty quickly:
tinyld-heavy --verbose "Notas para una aproximacin al punk espaol y sus contextos"
Analize chunks [ 'notas para una aproximacin al punk espaol y sus contextos' ]
[Pass 1] detectUniqueGrams 4-grams - match 'aol ' to ga
Merge Results [ { lang: 'ga', accuracy: 1 } ]
[ { lang: 'ga', accuracy: 1 } ]
I need to learn more about the library internals, and spelling mistakes will always be a problem, but is there anything worth tweaking?
We're using tinyld to detect the language on some external content, which generally works great. I'm looking to move to the heavy version to fix a detection error, but I noticed that one of our test cases started to fail. It's based on the title of https://www.scienceopen.com/hosted-document?doi=10.14293/S2199-1006.1.SOR-.PPL3VEC.v2, which contains a couple of spelling mistakes.
The normal version correctly identifies it as Spanish:
But the heavy version seems to decide it's Galician pretty quickly:
I need to learn more about the library internals, and spelling mistakes will always be a problem, but is there anything worth tweaking?