MT0031: Validate Word Segmentation Using the Unique Word List.

[jim-gyas]

Description:

Create a series of scripts to validate word segmentation by ensuring that each word in the target field:

• Exists in the unique word list.
• Is neither oversegmented (unnecessarily split into smaller parts) nor undersegmented (should have been split but wasn’t).

The process involves checking for both oversegmentation and undersegmentation issues, followed by merging the validated data to produce a final dataset that is free of segmentation errors.

Completion Criteria:

• Successfully validate word segmentation through multiple checks (valid word existence, oversegmentation, undersegmentation).
• Upload the final validated word segmentation dataset to Hugging Face in the required format, ensuring all entries meet platform requirements.

Implementation :

Generated Data Entries Info:

• Word Seg: 1535618.
• Simple Word Segement Validation: 1190714
• Oversegment word Validation: 66627
• Undersegment word Validation: 27130
• Final Validate word segment data: 8060

Subtasks:

• [x] Simple Word Segmentation Validation:

  • Create a script to validate word segmentation using the unique word list.
  • Check if each word in the target field exists in the word list. Mark entries as valid or invalid and list missing words for invalid entries.

• [x] Oversegmentation Validation:

  • Develop a script to detect oversegmented words, where a word is unnecessarily split into multiple parts.
  • Use the validated data from the previous task and check combinations of smaller tokens against the word list to see if they should form a single valid word. Mark entries as oversegmented when applicable.

• [x] Undersegmentation Validation:

  • Create a script to detect undersegmented words, where a word should have been split into multiple valid parts.
  • Use the validated data from the Simple Word Segmentation task, split words by the Tibetan syllable marker "་", and check each segment against the word list. Mark entries as undersegmented if valid segments are found.

• [x] Combine Non-Oversegmented and Non-Undersegmented Data:

  • Merge the results from the oversegmentation and undersegmentation validation scripts.
  • Take the intersection of entries that are neither oversegmented nor undersegmented as the final validated dataset.

• [x] Upload to Hugging Face:

  • Format the final validated data to meet Hugging Face’s requirements and upload it to the platform :https://huggingface.co/datasets/openpecha/standard_tibetan_word_segment

[kaldan007]

སྐུ་ ཞབས་ ལགས །

a c b c [a, b, ab ,c] ab c a b c ང་ཚོ་ མི་ཚེ འི་ ང་ཚོ་ ལ ས་ ཆུང་ཆུང་ཞིག་ གི་ མཚོན་བྱེད