search

apertium / apertium-apy

📦 Apertium HTTP Server in Python
https://wiki.apertium.org/wiki/Apertium-apy
GNU General Public License v3.0
32 stars 42 forks source link

Langid: Train for remaining languages that weren't in opus-100 #213

Open unhammer opened 1 year ago

unhammer commented 1 year ago

We have trained model files lid.beta.ftz and lid.release.ftz in the repo for languages that were in the opus-100 corpus. We should get corpora for the languages that weren't there and retrain (preferably in a fairly reproducible way, see scripts in ./ft-train).

Corpus suggestions: https://github.com/apertium/apertium-apy/pull/207#issuecomment-1398455482

Missing in release:

Got only 35791 lines for oci oc
Got only 35907 lines for sme se
Got only 67312 lines for bel be
Got only 6961 lines for arg an
Got only 79927 lines for kaz kk
No corpus found for crh
No corpus found for frp
No corpus found for szl
No corpus found for zlm

Full missing-list for beta and relase: https://github.com/apertium/apertium-apy/blob/master/ft-train/download-extract-corpus#L56

jonorthwash commented 8 months ago

How many lines of text do we need per language?

unhammer commented 8 months ago

The current script only uses the first 100.000 lines of text for each corpus. This was based on experiments with Scandi languages which can have very similar spelling (and then increased a bit) – if you have a language that is quite different from the rest of the set then I think you can get away with quite a bit less. As the above comment shows we "just" have 35k lines for sme (whereas e.g. deu has 100k), but https://beta.apertium.org/apy/identifyLang?q=ja+leat still gets it right