Add text segmentation for extended grapheme clusters - part 1

roc-lang / unicode

Universal Permissive License v1.0

7 stars 5 forks source link

Closed lukewilliamboswell closed 10 months ago

lukewilliamboswell commented 10 months ago

This PR

Set up the infrastructure to generate the internal modules for text segmentation using Unicode Character Database files
Includes a script to run code gen and test generated files from root
Includes most of the parser logic for parsing the code point and GBP from GraphemeBreakProperty-15.1.0.txt data file