MicrosoftDocs / typography-issues

Creative Commons Attribution 4.0 International
45 stars 21 forks source link

'kern' and 'palt' UI suggestions and feature interaction descriptions are confusing for CJK #1069

Closed macnmm closed 1 month ago

macnmm commented 10 months ago

OpenType ‘palt’ and ‘kern’ Documentation Revision Proposal

Nat McCully, Principal Scientist, Adobe

Introduction For CJK font developers, the wording in the documentation of the ‘kern’ and ‘palt’ features is confusing. This proposal aims to improve it so font developers will know how to define the features, and application developers will know how to set default behaviors for those features.

The Problem In the ‘kern’ documentation, it is stated:

UI suggestion: This feature should be active by default for horizontal text setting. Applications may wish to allow users to add further manually-specified adjustments to suit specific needs and tastes. Script/language sensitivity: None. Feature interaction: If 'kern' is activated, 'palt' must also be activated if it exists. If 'palt' is activated, there is no requirement that 'kern' must also be activated.

Meanwhile, the ‘palt’ documentation states:

UI suggestion: This feature would be off by default. Script/language sensitivity: Used mostly in CJKV fonts. Feature interaction: This feature is mutually exclusive with all other glyph-width features (e.g. 'fwid', 'halt', 'hwid', 'qwid' and 'twid'), which should be turned off when it’s applied. If 'palt' is activated, there is no requirement that 'kern' must also be activated. If 'kern' is activated, 'palt' must also be activated if it exists. See also 'vpal'.

There are several issues with this:

  1. The UI suggestions conflict with the feature interactions: ‘kern’ is on by default; ‘palt’ is off by default. However, when ‘kern’ is on, ‘palt’ must also be on. If ‘palt’ is off by default in CJK fonts, that implies ‘kern’ must also be off by default (at least for those glyphs affected by ‘palt’).
  2. The default options are complex: CJK fonts, containing both proportional Latin glyphs (which are, by default, kerned) and monospaced CJK glyphs (which can be set proportionally with ‘palt’ but not by default), seem to require several levels of default behavior. Which should be the default behavior, and is that default font-wide, or specific to certain ranges of the font’s glyphs? For example, when optionally setting CJK glyphs proportionally, should kerning also be on by default? Should kerning only be considered default for the non-CJK glyphs in a CJK font?
  3. When defining the mixed default behavior (kern only on where glyphs are proportional; off when they are monospaced), there is no standard method to discover which characters and glyphs or monospaced in a CJK font. One possibility is to use the UAX#11 East Asian Width property, and such can be suggested in the documentation of these features.

Improvement Suggestions I would improve the documentation with slight re-wording and addition of some more CJK-related information.

In the ‘kern’ documentation, change:

UI suggestion: This feature should be active by default for horizontal text in scripts other than mono-spaced CJK. Applications may wish to allow users to add further manually-specified adjustments to suit specific needs and tastes. Script/language sensitivity: By default shall not be activated on monospaced (e.g. CJK script) glyphs, but may be activated on any proportional (e.g. non-CJK) glyphs. Feature interaction: If 'kern' is activated, 'palt' must also be activated if it exists. If 'palt' is activated, there is no requirement that 'kern' must also be activated.

In addition to the above, we could consider adding a note explaining that CJK fonts often contain both proportional Latin and monospaced CJK glyphs, which makes kerning the glyphs more complex than in the case of an all-proportional font. When the font contains proportional glyphs, addition of kerning pair information would mean only those pairs with kern adjustments would be set proportionally, making the number of pairs likely to be very high and the cost of production very high as well. Using ‘palt’ first to transform the widths of the monospaced glyphs to be proportional and applying ‘kern’ deltas to a few pairs is much more sensible as a solution. Further, to determine whether a character is proportional by default, one can use the UAX#11 East Asian Width property, assuming this will encourage font foundries to assign monospaced or proportional glyphs according to the EAW in a consistent manner.

In the ‘palt’ documentation, change:

UI suggestion: This feature would be off by default. Script/language sensitivity: Used mostly in CJKV fonts to transform the monospaced glyphs’ widths to be proportional. Feature interaction: This feature is mutually exclusive with all other glyph-width features (e.g. 'fwid', 'halt', 'hwid', 'qwid' and 'twid'), which should be turned off when it’s applied. If 'palt' is activated, there is no requirement that 'kern' must also be activated. If 'kern' is activated, 'palt' must also be activated if it exists. See also 'vpal'.

In addition to the above, we could consider adding a note explaining that due to the conventional practice of including proportional Latin glyphs in the same font as CJK monospaced glyphs, and the fact that CJK text (e.g. Japanese text) can be set either monospaced or proportionally, and be kerned in addition to the base proportional widths for even finer tuning, that special consideration of the default values for such fonts is needed.

moyogo commented 9 months ago

@macnmm My apologies if I’m being confused for no reason, these suggestions do not seem clear and seem to assume CJK is monospaced.

If ‘palt’ is off by default in CJK fonts, that implies ‘kern’ must also be off by default (at least for those glyphs affected by ‘palt’).

The documention does not imply that, "would", "should", "must" are different. 'palt' must be on when 'kern' is on, 'kern' should be on by default, otherwise 'palt' would be off.

Which should be the default behavior, and is that default font-wide, or specific to certain ranges of the font’s glyphs?

The default should be 'kern' on, and if that’s the case 'palt' must be on, font-wide.

For example, when optionally setting CJK glyphs proportionally, should kerning also be on by default?

It depends what you mean:

Should kerning only be considered default for the non-CJK glyphs in a CJK font?

Either a font has kerning or not, for CJK or any group of glyphs, if it does the user may disable it. Why should 'kern' not be activated by default on CJK?

PeterConstable commented 4 months ago

The 'palt' feature (and 'fwid', 'pwid'...) were registered by Adobe. I suppose anyone who was involved when Adobe defined these is no longer there.

In the ‘kern’ documentation, change:

UI suggestion: This feature should be active by default for horizontal text in scripts other than mono-spaced CJK.

"Mono-spaced CJK" is not a script. While shaping engine implementations operating on per-script runs might have sets of default-on features to apply for a given script, I wouldn't expect them to make default feature application sensitive to other font-specific details.

On the other hand, an app (layered on top of shaping engines) could provide its own logic for application of discretionary features. More on that in a moment.


In general, I agree with @moyogo in expecting 'kern' to be activated by default for any script. Now, if a CJK font is mono-space, then it seems to me there wouldn't be any effect of a 'kern' feature. Presumably if a font is mono-space by default but has 'palt' to make it proportional, detecting that should be as simple as detecting that the font has 'palt'.

So, it seems it would be feasible for an app to detect when a run of CJK text is formatted with a font that has 'palt' and then turn 'kern' off by default for that run (whether or not the font implements 'kern').

Also, if a font does implement both 'kern' and 'palt', then I would assume the lookups triggered by 'kern' all come after those triggered by 'palt'.

If there are any apps that behave that way—detecting that a CJK run is formatted with a font that has 'palt' and disabling 'kern' be default'—and fonts designed to work with those apps that implement 'palt' and 'kern', then I'd expect the vendor that defined the 'palt' feature, Adobe, to have such apps and fonts.

Are there Adobe apps and fonts that behave that way?

macnmm commented 3 months ago

Added EAW suggestion.

macnmm commented 3 months ago

The 'palt' feature (and 'fwid', 'pwid'...) were registered by Adobe. I suppose anyone who was involved when Adobe defined these is no longer there.

In the ‘kern’ documentation, change: UI suggestion: This feature should be active by default for horizontal text in scripts other than mono-spaced CJK.

"Mono-spaced CJK" is not a script. While shaping engine implementations operating on per-script runs might have sets of default-on features to apply for a given script, I wouldn't expect them to make default feature application sensitive to other font-specific details.

On the other hand, an app (layered on top of shaping engines) could provide its own logic for application of discretionary features. More on that in a moment.

In general, I agree with @moyogo in expecting 'kern' to be activated by default for any script. Now, if a CJK font is mono-space, then it seems to me there wouldn't be any effect of a 'kern' feature. Presumably if a font is mono-space by default but has 'palt' to make it proportional, detecting that should be as simple as detecting that the font has 'palt'.

So, it seems it would be feasible for an app to detect when a run of CJK text is formatted with a font that has 'palt' and then turn 'kern' off by default for that run (whether or not the font implements 'kern').

Also, if a font does implement both 'kern' and 'palt', then I would assume the lookups triggered by 'kern' all come after those triggered by 'palt'.

If there are any apps that behave that way—detecting that a CJK run is formatted with a font that has 'palt' and disabling 'kern' be default'—and fonts designed to work with those apps that implement 'palt' and 'kern', then I'd expect the vendor that defined the 'palt' feature, Adobe, to have such apps and fonts.

Are there Adobe apps and fonts that behave that way?

Yes, all Adobe apps introduce a third default kerning method in the CJK versions called "和文等幅" or "Metrics - Roman only" which turns on 'kern' only for proportional Latin, and leaves 'kern' off otherwise. "Auto" or "Metrics" kerning activates both 'kern' and 'palt', so it will set CJK text proportionally and kerned as well as Latin, and is not recommended as a default for CJK.

macnmm commented 3 months ago

It was resolved in the AHG of MPEG-OTSPEC meeting of March 19, 2024 to move forward with these changes. I will work with @PeterCon on final wording.

PeterCon commented 3 months ago

@macnmm For starters, I could go ahead and add 'apkn' as I described on the mpeg-otspec list to OT 1.9.1, but I need to indicate who it was registered by. Although I provided the idea, I don't simply want to list Microsoft since I'm I haven't discussed with any of our product groups plans to implement. Can I add Adobe, or CITPC?

PeterCon commented 3 months ago

Cf. https://github.com/jcitpc/CJKFont/blob/main/docs/palt_kern.md

murata2makoto commented 3 months ago

The ISO Editorial Program Manager is much more strict than 3 or 4 years ago. The use of "must" for representing requirements is prohibited. "Must" is only for external constraints. More about this, see 7 Verbal forms for expressions of provisions in JTC1 Direcitves, Part 2.

PeterCon commented 3 months ago

Thanks, Murata-san. I'm fully aware of the ISO editorial requirements. When I submit any contributions to ISO to synchronize with content in the OpenType spec, I make that adjustment (including proposing such corrections to text previously published in OFF).

kidayasuo commented 3 months ago

Can I add Adobe, or CITPC?

I will double check with colleagues but I believe CITPC would not object being listed as a 'registered by' member.

kidayasuo commented 3 months ago

I believe our goals are:

  1. Fix the contradiction between ‘palt' and ‘kern’.
  2. Fix the ambiguity of “monospace” in the description of ‘palt' and ‘kern’, by defining it with or by suggesting to use EAW properties. It allows applications to implement ‘palt’, which is currently challenging because of the ambiguity.
  3. Define ‘apkn’ as a clean solution for new fonts. It frees applications from the ambiguity of monospace. Also it gives font designers flexibility of defining any character as fullwidth.

Do we all agree on these points? any additions / corrections?

kidayasuo commented 3 months ago

I thought moving to 'apkn' is a breaking change for fonts. It has an assumption that fonts would have to replace affected 'kern' entries with 'apkn'. Nat is wondering if it should be that way, i.e. fonts can leave corresponding 'kern' if they want to (@macnmm, please comment if necessary).

For applications that correctly support 'palt' and 'kern', if apps' and fonts' definitions of monospace code points match exactly, then, in theory, apps can be upgraded so that both old and new fonts behave identically, regardless of whether fonts retain duplicated 'kern' entries. If the definitions of monospace code points between apps and fonts do not align, then upgrading will result in a breaking change.

For applications that currently enable 'kern' unconditionally, removing duplicated 'kern' entries will be a breaking change. One problem I see is that if fonts do not remove duplicated 'kern' entries, supporting 'apkn' requires correctly ignoring 'kern' for code points that fonts provide 'apkn'. It is a pain even if doing so is at all possible, and dismisses the benefit of 'apkn'.

Therefore, I believe the corresponding 'kern' entries should be removed when 'apkn' is added. Applications should conditionally enable 'kern' only for fonts that include 'palt' but lack 'apkn'.

kidayasuo commented 3 months ago

Kerning between proportional and monospaced characters would need to be implemented through 'apkn', as kerning should not be enabled on such combinations by default, and 'palt' needs to be applied to the monospaced glyph.

kidayasuo commented 3 months ago

Can I add Adobe, or CITPC?

@PeterCon , actually CITPC wants to get feedback from member font vendors before becoming one of registerer (my apologies, I was too hasty!).

kidayasuo commented 3 months ago

Hello @PeterCon, will the palt & kern improvements be integrated into 1.9.1?

PeterConstable commented 3 months ago

@kidayasuo Yes, getting to that soon. Any feedback wrt CITPC?

kidayasuo commented 3 months ago

Super! Thank you! I would appreciate it if you could post or send me the drafts of 'palt'/'kern' so I can get feedback from CITPC folks. re: 'apkn' I still have to work at the CITPC side.

PeterCon commented 2 months ago

Not sure why I closed this earlier. Re-opening.

PeterCon commented 2 months ago

Draft text for the 'apkn' feature has been added to the OpenType 1.9.1 alpha:

https://learn.microsoft.com/en-us/typography/opentype/otspec191alpha/features_ae#tag-apkn

I'll work next on integrating with 'palt' and 'kern' descriptions.

PeterCon commented 2 months ago

@kidayasuo @macnmm Question for you folk: The issue raised focused on interaction between 'kern' and 'palt. In reviewing the many features Adobe registered for use in Japanese fonts, I'm wondering whether the issue raised doesn't also apply to a couple of other features:

The description for 'pwid' describes the difference between it and 'palt' as being only in how the feature is implemented: GSUB for 'palt', but GPOS for 'pwid'. (If that's the only difference, then there really wasn't a need to have separate features for the same typographic effect.)

Thoughts?

kidayasuo commented 2 months ago

@PeterCon The difference is that the 'palt' must be turned on when 'kern' is activated. This kern → palt feature interaction does not exist in 'pwid' or 'pkna'. They only have pwid/pkna → kern interaction.

With that said, I feel the requirement of turning kern on when pwid/pkna is activated is a strange one. Even if the spec suggests kern to be turned on by default, you can, and should be able to turn it off. @macnmm Shouldn't the pwid/pkna → kern interaction be removed?

fwid/hwid/twid/qwid deactivating kern would be reasonable.

I do not know why there are two similar features palt and pwid. @macnmm?

PeterCon commented 2 months ago

@kidayasuo I can revise wording about pwid/pkna → kern interaction. IIUC the interactions between all of these and kerning features (kern or apkn) should be basically the same, but currently the descriptions for the features are wording this differently. If one works well, the same pattern could be used in the other feature descriptions.

I've made draft revisions to the palt description, and also added a paragraph to the apkn description. (For some reason, the build actions didn't deploy the page update for the non-delta version of the page, so I'll provide a link to the delta page.) Please review:

takaakifuji commented 2 months ago

there really wasn't a need to have separate features for the same typographic effect

For the full-width Latin glyphs, some prefer to make them wider than the proportional ones. Still 'palt' and 'pwid' are usually expected to give the same effect for Kanji and Kana AFAIK. 'palt'/'vpal' is the only one that uses GPOS, and for the other width-related GSUB features, advance widths are baked into 'hmtx'/'vmtx'.

I agree that we should be able to turn kerning off for 'pwid' (and 'pkna') glyphs as Kida-san suggests – typically 'pwid' transforms e.g. U+FF21 FULLWIDTH LATIN CAPITAL LETTER A → U+0041 LATIN CAPITAL LETTER A. With 'pwid' and 'kern' activated, the Latin glyphs should be kerned by default, but originally-full-width glyphs should not. Thus I suppose kerning for 'pwid' glyphs should also be implemented in 'apkn' not in 'kern'. That might need a clarification a bit.

BTW for vertical writing I'm afraid we also need to introduce a tag which corresponds to vkrn. vapk maybe?

PeterCon commented 2 months ago

I've updated several feature descriptions to reflect addition of apkn and to use consistent wording:

I didn't mention pkna in the above descriptions ("mutually exclusive to..."), but I suppose it would make sense to add that.

I'll be updating the pwid, pkna and palt descriptions wrt interaction with kerning.

kidayasuo commented 2 months ago

Hell @PeterCon, regarding the 'palt' & 'kern' interaction, do you want to leave the ambiguity of the definition of the monospaced glyphs? We once talked about using EAW as an agreement between fonts and apps because checking if a specific glyph has 'palt' is not realistic performance wise. Considering all existing fonts use 'palt' we would want to clarify the expectation. thoughts?

PeterCon commented 2 months ago

I'm haven't yet updated kern and was planning to add something regarding that. I had to add apkn first, though, so started there, and the other places (with simpler changes) that need to mention it.

PeterCon commented 2 months ago

I tweaked the descriptions mentioned above (chws, etc.) to mention pkna as another glyph-width feature those are mutually exclusive to.

I also updated the description for pkna, and made changes to the description for apkn.

While working on this, it occurred to me that perhaps we might say apkn could be used to kern glyphs substituted by pkna or pwid. However, that would add the kind of complication we wanted to avoid: if palt and pkna/pwid are used in a font, then it's unclear what characters were affected by what without inspecting the lookup tables. So, I didn't pursue that.

PeterCon commented 2 months ago

And updated the pwid description along similar lines to the revision of pkna.

I'll work on updating kern tomorrow. Comments on the above are welcomed.

PeterCon commented 2 months ago

The kern description has been updated. I also tweaks some of the other descriptions.

We have not discussed hkna: it's not clear to me what "special horizontal forms" means, or how advance widths are affected by that feature. Addendum: I found examples that made clear how these two features are used. I've updated those feature descriptions to clarify.

kidayasuo commented 2 months ago

kern

This feature would not be used in monospaced fonts, and is not intended for use in vertical text layout. See the 'vkrn' feature for vertical kerning. …… Kerning adjustments should not be applied to fixed-width glyphs unless other features have been applied that adjust glyph metrics to be proportional.

An exception to default activation can apply for CJK text. If a text run is formatted with a font that implements the 'palt' feature, this feature should not be activated by default to glyphs covered by 'palt' lookups unless the 'palt' feature has been activated.

'chws': Application interface: If a layout engine supports advanced layout for CJK text as described in CLREQ, JLREQ or KLREQ, this feature should not be used. Otherwise, this feature should always be applied in horizontal layout of CJK text. …… It deactivates the 'kern' feature.

PeterConstable commented 2 months ago

Could the following text be interpreted to mean, 'Oh, so if I activate 'palt' by default, I can apply 'kern' unconditionally? That would be easier!' ?

That would be a valid interpretation. But this statement is in the "Recommended implementation" section, which I have always understood to mean guidance for font developers, not app developers. I'll revise the wording to be clearer.

A small matrix or flow chart might help understanding when apps can apply kerning for the whole text and when they can't.

I'm running out of time to add something like that, and I don't think it's strictly necessary for the specification of the features. All these features for Japanese that Adobe defined 25 years ago are confusing, and it would have been helpful if they had provided separate documentation, with matrices or charts, describing the relationships, differences and interactions between them.

It is likely a matter concerning 'chws'...

My understanding of chws, from the feature description, is that it results in glyphs that have fixed, half-width advances. If they are fixed width, then kerning shouldn't be applied. As for whether chws should be applied by default, I'll defer to Japanese experts such as you.

It deactivates the 'kern' feature.

~Argh!! I thought I got rid of all such wording: features cannot activate or disable other features. I'll revise that.~ I did get rid of all such wording. Please make sure you read the OT 1.9.1 alpha drafts (and refresh the pages, since I've been revising them today).

takaakifuji commented 2 months ago

could you confirm if EAW Ambiguous should be included here. I vaguely remember we discuss this once but forgot the conclusion

Kida-san and I did an analysis against the fonts in our library a month ago. We found Latin-1 Supplement and Latin Extended-A/B in those fonts were Ambiguous and the glyphs were designed as proportional, which means they expect to be kerned by default. Some of the Ambiguous glyphs had also 'palt', which means they were full-width, but the number was quite small compared to the ones that expect to be kerned by default. So we kinda concluded it makes less harm if Ambiguous is kerned by default. We'll have hits and misses depending on how fonts are implemented, but I suppose we should not explicitly state Ambiguous. Hope it makes sense!

PeterCon commented 2 months ago

I've updated the kern description to address comments above from @kidayasuo and @takaakifuji.

I've also added another new features, vapk, and revised the descriptions for vkrn and vpal, with smaller changes as well for valt, vchw and vhal.

For vkrn, the draft assumes that EAW property values are equally useful as a heuristic for exclusion from default vkrn activation as they are for exclusion from default kern application. Is that a valid assumption?

takaakifuji commented 2 months ago

Thank you so much for the updates!

Is that a valid assumption?

It sounds valid to me. For glyphs rotated by apps based on tables such as UAX#50, 'vkrn' won't be involved in the first place. For pre-rotated Latin glyphs which appear in 'vrt2', they are kerned to replicate the horizontal ones, so the assumption should work. For other upright forms including Kana or CJK punctuations, it should also work as they are mostly Wide or Fullwidth. I would be happy if Kida-san @kidayasuo confirms this.

The following are what I found:

kern

For instance, the glyph for U+FF08 “(” has full-width advance by default; the 'palt' feature adjusts the glyph to have a narrower, proportional width; this feature then shifts this glyph closer to U+3078 “へ” in the combination “(へ”.

すふ (U+3059 U+3075, Hiragana) or アノ (U+30A2 U+30CE, Katakana) would be a more natural example as a pair, which is often kerned tight. In Japanese an full-width open parenthesis is right-aligned and RSB usually won't be adjusted, and not that common to kern parentheses and letters unlike Latin. ぐ。 (U+3050 U+3002) would be another candidate as is more likely to have 'palt' than .

An application can scan the coverage tables of 'palt' lookups to determine glyphs to be excluded from default activation

For example in Kana, some (say or ) won't always have 'palt'/'vpal' adjustments because they are already wide/high enough and there are no room to reduce the sidebearings, but still they're kerned. So it could apply 'kern'/'vkrn' to a particular set of scripts/categories in an inconsistent manner. The basic idea should work but a better heuristic is needed to make it perfect.

Minor edits

PeterCon commented 2 months ago

@takaakifuji Thanks for the comments. I thought the example I had come up with probably was not very good, but wasn't sure what would be a good example.

Docs updated per the comments.

kidayasuo commented 2 months ago

Is that a valid assumption?

Yes, I agree with @takaakifuji. Characters that are fixed-width are fixed width regardless of the direction. There could be edge cases that I am overlooking but the use of EAW can’t be and does not need to be perfect.

Thank you @PeterCon

PeterCon commented 2 months ago

I think everything has been covered. See the OT 1.9.1 alpha for draft revisions addressing this issue and related issues discussed here.

kidayasuo commented 2 months ago

kern: This feature would not be used in fixed-width only fonts (no proportional features),

For this part do you mean so-called monospaced fonts like Courier, just to be sure?

PeterCon commented 2 months ago

Yes, that's what's meant. This statement was from the original description. I thought it was odd—it's obvious that a font like Courier wouldn't implement kerning, so the fact it's being stated makes you wonder if something else was in mind. And, it get's more convoluted given that there can be a font that is monospace by default but have features that give glyphs proportional widths (hence my attempt to cover that scenario). Maybe it would be better simply to delete that statement (from kern and vkrn).

PeterCon commented 2 months ago

I removed that statement from kern and vkrn as it is doesn't add anything important to the descriptions, is basically tautological and hence might only lead to confusion.

macnmm commented 1 month ago

We have found some Morisawa fonts that kern full-width glyphs with 'kern' only (not 'palt' and 'kern' together). Our stance is that by default neither 'kern' nor 'palt' nor 'palt + kern' should be applied to Japanese characters. If the user is allowed to distinguish between kerned (in our case this means 'palt' + 'kern' always, and if there is no 'palt' no harm done), and proportional widths ('palt' only), that is best. So, "kerned" and "proportional" can be differentiated by the user and neither should be on by default. For non-Japanese text, "kerned" (in our case 'palt' + 'kern') text is default.

PeterCon commented 1 month ago

@macnmm It's not clear from your comment if you think any further change is needed. If not, then I'd like to close this issue.

See the OT 1.9.1 beta for draft revisions addressing this issue.

PeterCon commented 1 month ago

No further feedback received, so closing. (Getting ready to release OT 1.9.1.)