kidayasuo
commented
3 years ago Eric, thank you very much for your proposal.
I have several comments / questions regarding your proposal. I would greatly appreciate it if you could clarify.
grapheme cluster vs grapheme cluster occurence
What is a difference between A and A occurence in this context? (may be this is simply a novice English question)
it distinguishes the class used in horizontal and in vertical texts
What is the benefit of having different classes between horizontal and vertical? I would appreciate it if you could elaborate here a bit.
The locale method has the downside that authors are not always tagging their text appropriately (either not at all, or not carefully on punctuation).
I could not figure out what “not carefully on punctuation” means…
but it also means that one can specify different glues for e.g. ideographic/inseparable_emDash and ideographic/inseparable_twoDotLeader, or specify different glues for inseparable_emDash/inseparable_twoDotLeader and inseparable_emDash/inseparable_ellipsis.
I can see why separating each inseparable makes sense, because that way you can construct a state machine. Regarding your second reasoning, i.e. ability to define different glues, do you have concrete examples of why such ability is a good thing?
As noted earlier, markup should always be available to influence the determination of the glue. Thus there is no need for such a Unicode property to be perfect; it does however need to be easily accessible and fairly stable.
I am not sure if I understood the discussion here. There will be systems / applications where such markup is not possible (due to limitation of the underlaying engine, or limitation of the UI), and if so, existence of a markup could not be the reason why the property does not need to be perfect. I am not necessarily saying it needs to be perfect, as it can’t be.
The classes are only one part of the final visual appearance: the glue settings also come into play, so it is worth discussing those a bit, as they may influence the design of the classes.
Do you have examples of the glue design giving influence on the design of the character classes?
I think the best way forward is to recommend that for paragraphs using the spacing model discussed here, that glue be controlled by the spacing model (i.e. the mojikumi tables) and that text-indent be set to 0.
Do you suggest that there will be a part of text where this model is applied and another part of text where it is not applied, instead of defining everything in one spacing model? If you have multiple models, especially in one document, it would become harder to for example set an uniform style, e.g. indentation.
Columns: I have not yet carefully looked at the character class assignments you proposed but how did you handle JLReq classes that are dependent on the layout, such as ruby base (cl-20, 21, 22, 23, 24, 25, 28, 29, 30)?
0x000021 | R | H | westernChar 0x000080 | R | H | V | unknown
Is it correct to interpret that code points between 0x000021 and 0x000080 follow properties specified by 0x000021?
Lastly it would be great if you could provide a cross reference between JLReq classes and classes in your proposal.
Thank you!
emuller-amazon
Eric Muller posted a proposal on the admin list. I am copying it here to track discussions.
a sub issue of: #240 ––––––––––––––––––––––––––––––––––––––––––––––
2020/10/19 8:23、Eric Muller emuller@amazon.com wrote:
Here is our perspective as implementers. It is a bit raw (sorry, we noticed the announcement a bit late), don't hesitate to reach out for clarification.
Eric.
Character classes serve two purposes: linebreak opportunities and spacing around characters.
Linebreak opportunities are adequately handled by Unicode currently, at most needing some adjustment in UAX14 or in the CLDR language tailorings. Therefore that use is not discussed here.
A possible spacing model is that there is glue (variable space) on each side of each grapheme cluster occurrence. This glue is characterized by its natural width (JLREQ appendix B) and can be deformed (either compressed - JLREQ appendix D - or expanded - JLREQ appendix E) to achieve justification.
While each glue occurrence could be specified explicitly via markup, it can be determined most of the time from its context, using classes: for a left glue, by the class of what's on the left of the grapheme cluster occurrence and by the class of the grapheme cluster occurrence itself; and similarly for a right glue, by the class of the grapheme cluster occurrence and by the class of what's on the right of the grapheme cluster occurrence.
What's on the left (or right) of a grapheme cluster occurrence may be another grapheme cluster occurrence, in which case the class of "what's on the left" is the class of that other grapheme cluster occurrence. But it can also be that there is no other grapheme cluster occurrence on the left, or there is some intervening graphical element, thus leading to classes:
The class of a grapheme cluster occurrence could also be specified explicitly by markup, but it can often be determined from the characters composing the grapheme cluster occurrence (at which point, it is the same for all occurrences of a given grapheme cluster). That can in turn be determined from classes assigned to the characters in the grapheme cluster. Generally, the base character of a grapheme cluster determines the class of the grapheme cluster, but there are cases where the other characters "dominate" the determination: for example, <U+00A0 NO-BREAK SPACE> may be in a class, and <U+00A0 U+0301 COMBINING ACUTE> may be in a different class.
Finally, we arrive at the classes of characters. Below is a proposed assignment for the whole Unicode repertoire. This classification mostly aligns with that of JLREQ, with a few differences:
for unassigned code points (in the Unicode sense), the class is a prediction based on the likely future allocation of those code points
JLREQ simply ignores the existence of the full width characters at U+FFxx. This leads to a number of "ambiguous" characters, such as U+0041 LATIN CAPITAL LETTER A, where JLREQ says both "an occurrence of U+0041 could be in the Western class" (A.27) and "an occurrence of U+0041 could be in the Ideographic class" (A.19). In practice, authors routinely use U+0041 and U+FF21 precisely to disambiguate the class to use.
it distinguishes the class used in horizontal and in vertical texts
it distinguishes the inseparables (see below)
it uses the InDesign refinement of the opening and closing classes (square, rounded, other)
The proposed assignment also mentions the UAX50 vertical orientation property, as it is closed aligned and informs the spacing class assignment.
Ambiguous characters
While most characters are unambiguously in a class, regardless of their context, a few characters common in Japanese typography are inherently ambiguous:
A possibility is to resolve those based on the locale, or their resolved script (itself determined by looking at the script of the adjacent character).
The locale method has the downside that authors are not always tagging their text appropriately (either not at all, or not carefully on punctuation).
The script method has the advantage of not requiring the author's help, and that computation is already necessary in OpenType layout engines.
Inseparables
Currently, all inseparables are lumped in a single class, and a footnote explains that the behavior inseparable/inseparable applies only to two occurrences of the same inseparable. It would be better to have separate classes for inseperables. Not only does that avoid a footnote, but it also means that one can specify different glues for e.g. ideographic/inseparable_emDash and ideographic/inseparable_twoDotLeader, or specify different glues for inseparable_emDash/inseparable_twoDotLeader and inseparable_emDash/inseparable_ellipsis.
Logical vs visual order:
It should be made clear that the practical definition of glues is in the visual space: that's why we used the terms "left" and "right".
Classes as a Unicode property
From a practical point of view, I believe that the spacing class should be part of the Unicode Character Database, as a property, just like the vertical orientation property. The main reason is that this is the most reliable way to get a something well defined (in the sense of having a definition, not necessarily in the sense of having correct values), and in sync with the Unicode repertoire. It is a relatively easy task for Unicode, as has been demonstrated with the vertical orientation property. (In fact, the very first draft of what because UAX50 included the spacing class).
It is worth nothing that such a Unicode property is only a starting point. As noted earlier, markup should always be available to influence the determination of the glue. Thus there is no need for such a Unicode property to be perfect; it does however need to be easily accessible and fairly stable.
======== Classes and glue settings
The classes are only one part of the final visual appearance: the glue settings also come into play, so it is worth discussing those a bit, as they may influence the design of the classes.
Glue settings and justification
When justifying text (a common case for body text), implementation may have to expand a glue to an arbitrary width. Consider for example a two character paragraph, with text-align-last: justify, the glue has to be (linewidth - 2em). While large glues are sometime the result of pathological conditions, they can also be explicitly intended, such as in jidori processing. Thus it is desirable to allow pretty much all glues to grow to indefinitely.
Glue settings are mostly for body text
JLREQ currently describes three glue settings (default, JIS, and book, in tables 3-5 of appendix D; they differ only on the behavior when compressing lines, but in principle different settings could also differ on natural width or when expanding lines). It seems that those setting are mostly concerned with body text, and are not appropriate for, e.g., titles. For example, the default method specifies 0 glue between paragraph (line) start and an opening bracket, and 0.5em between a closing bracket and a paragraph (line) end; for a title starting and ending with brackets, which happens to be set on two lines (centered and not justified), this assymetric can be jarring.
It would be worth having a discussion that the settings apply to body text and to mention when they are not appropriate, or even better to include setting for other other situations. The most important situation that come to mind: titles, and ruby base/ruby text.
Interchange of glue settings
The discussion so far has been about determining the classes from characters, leaving room for document styling systems (e.g. CSS) to let authors explicitly specify classes of occurrences. The classification is of course only one part of the final result, the other being the glues that result from those classes (i.e. JLREQ appendices B, D, E). It would be useful to encourage document styling systems to allow the specification the glues as well, in the documents, either in the form of selecting from a predetermined set of settings, or by completely specifying the settings (may be as delta on top of the predetermined settings).
Spacing classes and the CSS text-indent property.
With the model presented above, the CSS text-indent property is essentially an unconditional, invariable glue between to the left of the first grapheme in a paragraph. In practice, it is useful in Japanese typography to make that glue at least conditional: e.g. 1em before an ideograph, and 0.5em before an opening bracket. I think the best way forward is to recommend that for paragraphs using the spacing model discussed here, that glue be controlled by the spacing model (i.e. the mojikumi tables) and that text-indent be set to 0.
========
Columns:
A: if the resolved script is not Hans, Hant, Jpan -> westernChar
===========