christianparpart
commented
3 years ago Thanks for pointing me to this ticket @jerch.
I have been actually implementing grapheme cluster support in my terminal emulator. My motivation behind all the pain was to be able to properly render complex emoji in the terminal. Now I'd like to address some of your points above:
apps compiled with older unicode version in mind will disagree about the runwidth of certain characters (down to not supporting graphemes at all).
I think that's not a problem, because in non-supporting client apps it will be highly unlikely that that application will render complex Unicode anyways.
Unicode is highly uncertain about monospaced environments
This is right, but that should not affect grapheme cluster aware terminals. In my opinion. In the context of a terminal, historically every displayed character usually was 1 grid cell wide. And because of that, recent terminals that started to support displaying emoji did indeed render emoji over 2 grid cells, but did not increment the cursor by 2 cells to not break existing applications. I think that wasn't that bad, at least we had emoji. But most terminals are not grapheme cluster aware (seriously, are there any?)
Now, grapheme clusters can be used to determine how many consecutive codepoints should be rendered as "one user perceived character". Characters on the terminal are usually in one grid cell, and a few in 2 grid cells for wide characters (such as emoji, kanji, ...). Implementing grapheme cluster segmentation in a terminal would help putting the right "non-breakable" sequence of codepoints into one grid cell - just like the user would expect them to be (as it's one user perceived character when rendered). I'd highly vote for supporting that. :)
Complex graphemes might extend character dimensions in both directions, which might break the grid (characters might get cut in height and width).
AFAIR, the first codepoint mandates the EastAsian width unless VS15/VS16 is involved which explicitly states to enforce the emoji presentation to either text (VS15, narrow, 1 character cell) or emoji (VS16, wide, 2 char cells).
Grapheme clusters are not all out of the suddon making text look vertical (ref to "both directions"). I think not even mlterm is doing that. Even RTL is a hard (but apparently solved) problem (again: mlterm, gnome-terminal to some degree?). But I think either should hinder properly interpreting grapheme cluster boundaries.
Unicode explicitly allows different valid output representations of compound chars/emojis, depending on the render system whether it can create a compound symbol or not
I think what you are referring to is that for example the family emoji of course usually looks like a family, but some applications might render it as a sequence of emoji symbols. Or a colored person emoji might alternatively rendered as standard (I think: white) colored emoji with the color modification emoji symbol right next to it.
The quote from the UTS 51, section 2.2 is as follows:
However, if that combination is not supported as a single unit, it may show up as a sequence like the following, and the user sees no indication that it was meant to be composed into a single image:
To me, that is not as bad as it sounds, unless one plans to intentionally implement a TE that does fall under the above mentioned category of "not supported". Everyone should at least attempt to support and that should not hinder grapheme segmentation boundary determination in a TE (IMHO).
I hope this is not too much of a wall of text. One last note. The grapheme cluster segmentation algorithm might be expensive to execute, especially in the context of a terminal, where some people like to perform terminal output bandwidth performance tests recently. I found myself in a weak spot and realized that the algorithm I implemented was naively following the rule-set as specified in https://unicode.org/reports/tr29/#Grapheme_Cluster_Boundary_Rules . However in the context of a TE this algorithm can be perfectly optimized for the standard TE case, so that only for the actual non US-ASCII codepoints the algorithm has to be fully run (my performance win was 50%! of total performance in my TE bandwidth benchmark!)
p.s.: In case legacy applications might still be of concern to some, we could agree on a feature detection number that could be queried by client apps that would want to distinguish. The user could make that feature per config default enabled/disabled and an SM/RM code could be even introduced to allow apps to decide whether or not grapheme segmentation should be obeyed. (Even though, personally I'd highly advocate to always respect grapheme clusters).
jerch
Issue to track grapheme cluster and follow-up unicode version support.
Currently we are not grapheme cluster aware which leads to several issues around complex combining unicode characters like compound emojis and scripting systems. Final goal should be to handle most newer aspects of Unicode 11+ with a dedicated v13 addon.
TODO
wcwidthhandling inInputHandler.printwith grapheme rulesetbonus goals
Limitations While support for graphemes and v13 will solve several output issues for compound characters on newer OS, it certainly will not solve all unicode related issues:
useCompoundUnicodeRenderWidth = True) to tell the renderer always to use the real available glyph width. The runwidth ambiguity for app side here could be solved with an additional terminal sequence requesting the real runwidth prior usage (e.g. a chat app that knowns its used compound emojis can request all runwidths before entering the main texting loop).