Checking the source dictionary

jaumeortola commented 7 months ago

I will be adding some comments and ideas here. We can open new issues for some parts of the work.

We can proceed to separate the entries by groups: the ones that don't need review, the ones that need some manual review, and so on. For example:
- words in all spelling dicts and tagged -> no need to review
- words in all spelling dicts but not tagged -> maybe they can be tagged easily
- words in US and GB and tagged -> maybe they can be accepted by all variants?
- ...
Check that variant labels are coherent with en-US-GB.txt (use scripting).
Some sets of entries look suspicious: untagged words in GB with some prefixes (mis-, out-, over-, re-, under-) seem nonsense words. The same with some affixes (see: survivorshipably... survivorshipry).
Words with the tag us-large come from a Hunspell US dictionary that we didn't use until now. It is mentioned in https://github.com/languagetool-org/english-pos-dict/issues/2
We are using a simplified format for regular verbs: recharge=verb=all. (We use a few rules to cover more cases of regular verbs. See here). It would be useful to have something similar for nouns: a simple and quick way to tag a noun. We would need to define the format, and ways to write exceptions.
What sources we consider authoritative to determine if a word is GB or US? And AU, CA, ZA, NZ? Are there dictionaries for those variants?

jaumeortola commented 7 months ago

Separating the dict in src-clean.txt (accepted entries), src-pending.txt and src-discarded.txt. https://github.com/languagetool-org/english-pos-dict/commit/551de86c684b157bce7b4842d636ddb3bfb0238b

Done:

[x] tagged words in all spelling dicts -> moved to src-clean
[x] tagged words in GB and US -> moved to src-clean, variant modified to "all"
[x] untagged words in GB with some prefixes (mis-, out-, over-, re-, under-, in- + uppercase) -> moved to src-discarded

AzadehSafakish commented 6 months ago

Next steps, perhaps in this order(?)

languagetool-org / english-pos-dict