Closed lukhnos closed 3 months ago
Thank you so much for this change! This will make the dictionary way easier to maintainable. Let's also update the wiki once this is merged [1]
@lukhnos I've tried to read the diffs but too complex to be consistently verified, so I just approve this and then make a plan of reimplementing an algorithm to verify it systematically.
Sorry to make this too complex. Here the diff against the current phrase.occ
—first sorted with LC_ALL=c sort
and all tabs replaced with single spaces—against the one in the PR: diff. There are 26 entries removed, and that's how I discovered that those dups had contributed to issues such as "電子" might not always come before "墊子".
I'll merge this now, and hopefully after the sorting we'll have an easier time reviewing these files.
@lukhnos I've tried to read the diffs but too complex to be consistently verified, so I just approve this and then make a plan of reimplementing an algorithm to verify it systematically.
Sorry to make this too complex. Here the diff against the current
phrase.occ
—first sorted withLC_ALL=c sort
and all tabs replaced with single spaces—against the one in the PR: diff. There are 26 entries removed, and that's how I discovered that those dups had contributed to issues such as "電子" might not always come before "墊子".I'll merge this now, and hopefully after the sorting we'll have an easier time reviewing these files.
@lukhnos No worries at all and thank you so much for the clarification!
Let's also update the wiki once this is merged
增加了一節 https://github.com/openvanilla/McBopomofo/wiki/詞庫開發說明#請確保-bpmfmappingtxt-以及-phraseocc-兩個檔案的詞條排序
Fixes #447
This also dedups phrase.occ and fixes a few issues with phrases such as 電子 (vs 靛紫) due to the inadvertent dups having much lower scores.
phrase.occ
sorted and deduped with a one-time script: https://gist.github.com/lukhnos/2d4c628e2690b5407777f2b75f699a89cc @xatier