jquast / wcwidth

Python library that measures the width of unicode strings rendered to a terminal
Other
393 stars 58 forks source link

Variation Selector 15 (VS-15, U+FE0E) support. #120

Open jquast opened 7 months ago

jquast commented 7 months ago

I did a few spot checks of VS-15 when implementing VS-16, and erroneously believed that all emojis in VS-15 sequences were already listed as by EastAsianWidth.txt as width of 1.

But that's not true. There are several emojis that are "wide" that are changed to "narrow" with VS-15.

Reported by @rivo in https://github.com/muesli/reflow/issues/73#issuecomment-1944274659

@rivo: you declare that our "Specification" is "missing some things", I would appreciate any further things that you find wrong.

codecov[bot] commented 7 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (056ee4b) 100.00% compared to head (f00fba5) 100.00%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## master #120 +/- ## ========================================= Coverage 100.00% 100.00% ========================================= Files 5 6 +1 Lines 105 115 +10 Branches 25 28 +3 ========================================= + Hits 105 115 +10 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

rivo commented 7 months ago

@rivo: you declare that our "Specification" is "missing some things", I would appreciate any further things that you find wrong.

I guess VS15 was the main one. I noticed other minor things like U+FF9E or U+FF9F which the grapheme cluster spec lists as "extending characters", thus width of 0 IMO but would get a width of 1 according to your algorithm, thus, e.g. ギ would result in a width of 2.

I'm also not sure what exactly this means:

Any character following ZWJ (U+200D) when in sequence by function wcwidth.wcswidth().

How do you determine when such a sequence ends? E.g. how do you determine the width of a string such as this one:

👩‍👩‍👦‍👦abc

I personally also think that characters such as U+2E3B should be much wider (width of 1 in your library). I've seen it span at least 4 cells in various macOS applications.

There may be more, or maybe not, I don't know. Validating your algorithm would be a lot of effort.

I put the word Specification in quotes because while you do note that it's a description of your implementation, I find that writing "I authored a formal Specification detailing how characters should be measured" makes it sound more authoritative than it is, especially in the context of assigning grades to other applications based on how well they "perform". In my comment, I was trying to make the point that there is no official specification and so far, every attempt I have seen has had flaws.

Even my own implementation is not perfect. Consider U+FDFD which both of our libraries will assign a width of 1:

U+FDFD: ﷽

In iTerm2, it spans 5 cells. In Chrome on macOS with a monospace font, it's even 11 cells wide.

It would be interesting to render out these characters on different platforms with the most common fonts and check how their widths compare to our calculated widths. This might reveal some general flaws or at the very least it would identify outliers such as U+FDFD.

I haven't had time for such a project yet, though.

GalaxySnail commented 7 months ago

I noticed other minor things like U+FF9E or U+FF9F which the grapheme cluster spec lists as "extending characters", thus width of 0 IMO but would get a width of 1 according to your algorithm, thus, e.g. ギ would result in a width of 2.

Shouldn't the width of ギ be 2? This string is aligned with East Asian Wide characters in my web browser.

ギ一二三
一二三四

rivo commented 7 months ago

Shouldn't the width of ギ be 2?

This is U+FF77 and U+FF9E. U+FF77 is a half-width character, thus width=1. Is U+FF9E a separate character? The Grapheme Cluster Spec says it's an "extending character" and those typically don't take up extra space. They typically fall into the "Mn" category.

I'm aware that some fonts will still create extra space for them, which is likely why it looks ok in your browser. I guess it's nothing big to worry about. (That's why I wrote "minor" above.)

jquast commented 7 months ago

Thank you for your feedback @rivo I honestly appreciate it,

Any character following ZWJ (U+200D) when in sequence by function wcwidth.wcswidth().

How do you determine when such a sequence ends? E.g. how do you determine the width of a string such as this one:

👩‍👩‍👦‍👦abc (U+1f469, U+200d, U+1f469, U+200d, U+1f466, U+200d, U+1f466, 'a', 'b', 'c')

The characters that follow U+200d are not counted. So the first character, U+1f469 is of width 2, but the characters following the three U+200d's (U+1f469, U+1f466, U+1f466) are not counted, while 'abc' is counted normally. So the final width is 2 + 3 = 5:

>>> import wcwidth
>>> wcwidth.wcswidth('👩‍👩‍👦‍👦abc')
5

The algorithm in wcswidth is fairly basic, https://github.com/jquast/wcwidth/blob/056ee4ba0df66fb33be535d8f37470685ef32ba9/wcwidth/wcwidth.py#L189-L192

There may be more, or maybe not, I don't know. Validating your algorithm would be a lot of effort.

Maybe you missed the details of the ucs-detect tool that I have written, but it does validate the ZWJ algorithm is 100% compliant with Konsole, foot, iTerm2, and WezTerm. (A+ score for "ZWJ" at https://ucs-detect.readthedocs.io/results.html)

I find that writing "I authored a formal Specification detailing how characters should be measured" makes it sound more authoritative than it is

My apologies for that. I will modify all references to be very specific that it is the "Specification of how python wcwidth package measures..." etc, I can only say that I try to be as terse as possible. Because wcwidth is of interest to non-english speakers that may be reading it with difficulty or through translation, I try not to mince too many words, so it may come across as more authoritative than I intended.

U+FDFD: ﷽

In iTerm2, it spans 5 cells. In Chrome on macOS with a monospace font, it's even 11 cells wide.

As for these kinds of scripts (Arabic in this case), fixed-width fonts and monospace constraints of a terminal is not appropriate. We can only do so much to interpret unicode.org specifications for the terminal environment, but measuring this kind of script isn't possible with the data files that they publish. Supporting this kind of thing would require digging into the font and its rendering engine, which wouldn't be very reasonable to implement for a general purpose command-line application library. Even terminal emulators don't often dig into the font engine. I don't believe that folks who use these languages will be very successful in designing interactive curses applications.

Aside, I do wish for there to be a terminal sequence to display variable-width fonts and be released from the constraints of monospacing for such languages. Such a sequence could be used or detected at the application level to assume that the position is indeterminate, and rely on "cursor position report" queries for only an approximation of the nearest current cell.

Because popular multi-language terminals (mlterm, foot, iTerm2, Konsole) measure it as width of 1 then that is what I wish for my library to report. I don't wish to invent any new specification or standards, I apologize if it is ever interpreted that way, I will include more phrasing in the README.rst to make that clear. For example, if I found a statement in a unicode.org document that disagreed with all popular terminals, I would rather our library and specification match the most popular terminals.

rivo commented 6 months ago

I appreciate your thoughtful response.

monospace constraints of a terminal

Our goals may differ but in my Golang implementation, I don't think I mention terminals at all. Of course, it's a common area where these libraries are being used. But, for example, lots of people use VS Code and other IDEs which also use monospace fonts (except for the few who swear by variable fonts for programming but let's ignore them for now). Increasingly, such editors are used for more than just programming. Markdown, for example, appears to be integrated in more and more contexts (e.g. blog publishing, note taking, e-book authoring) and since IDEs have good support for writing Markdown, people will naturally gravitate towards using them. So I expect that these kind of algorithms are / will be relevant outside of terminals and for many different languages.

You are completely right in that there is value in offering an algorithm that matches the most commonly used terminals, even when they're "wrong". There is no point in deciding a character is 2 cells wide when all terminals render it in 1 cell. In tview, my terminal UI library, I can say ﷽ is 5 cells wide but when it gets rendered out to the terminal, I need to assume the terminal assumes it to be 1 cell, therefore I have to output another 4 space characters to match my own interpretation. So I need both my own width library and the wcwidth library which matches most terminals.