Closed eggrobin closed 1 day ago
@makotokato @aethanyc
@aethanyc or @makotokato can you take this issue? Probably for 1.x Priority.
Discussion: Longer term, we would like it if the upstreamed TOML files would be updated along with the specification, so that ICU4X does not need to do anything more than pulling in updates from upstream.
Looking at the toml files, my impression is that they define a state machine transitioned by code point (that is, a [[tables]]
record defines a transition from its left
state to its name
state when the next code point has the class right
), and that the breaks at each step are determined by the [[rules]]
with a matching left
state, and looking ahead one code point matching the class right
.
The following new line breaking rules require more lookahead than that:
× [\p{Pf}&QU] ( SP | GL | WJ | CL | QU | CP | EX | IS | SY | BK | CR | LF | NL | eot)
(AK | ◌ | AS) × (AK | ◌ | AS) VF
These require looking at two code points to the right of the (non-)break, plus any intervening CM (since these are after LB9).
Henri, this is interesting.
In your comment you correctly identified what LB15a and LB15b are trying to do, and why they need to do that (instead of treating Pi as LB=OP and Pf as LB=CL: that would mess with German, Finnish, etc. usage of Pf initially or Pi finally).
However, these new rules do not help with the Chinese issue at hand, since there are no spaces (there may visually appear to be space, but that is because U+2018 etc. have ambiguous width; here they are wide). This has recently come to the attention of the Properties and Algorithms Group of the UTC; it may be possible to do something about it in the ID QU ID case. I will mention that issue in that discussion. Nothing will happen on that front before Unicode 16.0 in September 2024 though.
We still need to update line segmenter to Unicode 15.1. @makotokato is working on it.
I am experimenting with moving LB8a and LB9 into the code of the line segmenter, as
The Properties and Algorithms Group plans to recommend the following proposals to Unicode Technical Committee #175 later this month. If they are accepted, the changes would be published as part of Unicode Version 15.1, in September.
UAX #14:
UAX #29:
LinkingConsonant ExtCccZwj* Virama ExtCccZwj* × LinkingConsonant
, where:Virama=[\p{Gujr}\p{sc=Telu}\p{sc=Mlym}\p{sc=Orya}\p{sc=Beng}\p{sc=Deva}&\p{Indic_Syllabic_Category=Virama}]
LinkingConsonant=[\p{Gujr}\p{sc=Telu}\p{sc=Mlym}\p{sc=Orya}\p{sc=Beng}\p{sc=Deva}&\p{Indic_Syllabic_Category=Consonant}]
ExtCccZwj=[\p{gcb=Extend}-\p{ccc=0}] \p{gcb=ZWJ}]