Make opentype tables compatible with Universal Shaping Engine for scripts routed to USE

[jungshik]

Spun off from notofonts/noto-fonts#543, notofonts/balinese#22, notofonts/lepcha#3, and notofonts/tifinagh#12.

We need to make Phase 3 fonts for scripts that are routed to USE be USE-compatible.

https://www.microsoft.com/typography/OpenTypeDev/USE/intro.htm

/cc @roozbehp @tiroj @marekjez86 @behdad @TrueTyper @waksmonskiMT @BrianKraimerMT

[jungshik]

Scripts that go to USE (Universal Shaping Engine) are listed below. (taken from @behdad's comment : https://github.com/googlei18n/noto-fonts/issues/543#issuecomment-153506926 )

Balinese
Batak
Brahmi
Buginese
Buhid
Hanunoo
Kaithi
Kharoshthi
Lepcha
Limbu
MeeteiMayek
Rejang
Saurashtra
Sundanese
SylotiNagri
Tagalog
Tagbanwa
TaiLe
TaiViet

/cc @jamesgk

[jungshik]

@waksmonskiMT

[jungshik]

/cc @kmansourMT

[xiangyexiao]

We expect to get the fix by end of April. The early the better. Is it feasible?

[kmansourMT]

Xiangye, please clarify:

What fix will be ready by end of April?

To whom is the following question directed: Is it feasible?

From: Xiangye Xiao notifications@github.com<mailto:notifications@github.com> Reply-To: googlei18n/noto-fonts reply@reply.github.com<mailto:reply@reply.github.com> Date: Thursday, 25 February 2016 at 15:28 To: googlei18n/noto-fonts noto-fonts@noreply.github.com<mailto:noto-fonts@noreply.github.com> Cc: Kamal Mansour kamal.mansour@monotype.com<mailto:kamal.mansour@monotype.com> Subject: Re: [noto-fonts] Make opentype tables compatible with Universal Shaping Engine for scripts routed to USE (#576)

We expect to get the fix by end of April. The early the better. Is it feasible?

— Reply to this email directly or view it on GitHubhttps://github.com/googlei18n/noto-fonts/issues/576#issuecomment-189033563.

[xiangyexiao]

sorry for the confusion. We (Jungshik) already fixed the bugs for Phase II fonts. The timeline in my last comment is for ourselves to update Android with the new fonts.

[xiangyexiao]

still assign to Sue since this bug is for Phase III.

[jungshik]

It cannot be priority critical. This should happen as Phase III fonts are being developed.

[roozbehp]

You're correct Jungshik. I failed to notice that the bug is about Phase 3 fonts.

[nizarsq]

@marekjez86 what is the status? suggesting on how I should create test case?

[behdad]

@marekjez86 what is the status? suggesting on how I should create test case?

Close this as Monotype refused to do even though they contractually should have, and no one else has bandwidth to do it now. People will complain if noto fonts are broken and we go from there.

[punchcutter]

Most scripts were updated for phase III when USE was actually getting implemented. I only see a few now that don't have the script specific tag and only use DFLT. For those I only know the reason for Zanabazar Square as stated below. Elymaic I don't know. Tifinagh is a question for JamraPatel since they did it. Tai Tham has been in discussion for a long time about how to handle it in USE so it's not really working 100% anyway. So I think this issue should definitely be closed and new issues can be opened if there's an issue with any particular script.

ADLaM ✅ Ahom ✅ Bhaiksuki ✅ Balinese ✅ Batak ✅ Brahmi ✅ Buginese ✅ Buhid ✅ Chakma ✅ Cham ✅ Chorasmian - no font Dives Akuru - no font Dogra ✅ Duployan - no features, no script tags Egyptian Hieroglyphs - no features, no script tags Elymaic ❌ Grantha ✅ Gunjala Gondi ✅ Hanifi Rohingya ✅ Hanunoo ✅ Javanese ✅ Kaithi ✅ Kayah Li ✅ Kharoshthi ✅ Khitan Small Script - no font Khojki ✅ Khudawadi ✅ Lepcha ✅ Limbu ✅ Mahajani ✅ Makasar - no font Mandaic ✅ Manichaean ✅ Marchen ✅ Masaram Gondi ✅ Medefaidrin - no font Meitei Mayek ✅ Miao ✅ Modi ✅ Mongolian ✅ Multani - no features, no script tags Nandinagari - no font Newa ✅ N’Ko ✅ Nyiakeng Puachue Hmong ✅ Old Sogdian ✅ Pahawh Hmong ✅ Phags-pa ✅ Psalter Pahlavi ✅ Rejang ✅ Saurashtra ✅ Sharada ✅ Siddham ✅ Sinhala ✅ Sogdian ✅ Soyombo ✅ Sundanese ✅ Syloti Nagri ✅ Tagalog ✅ Tagbanwa - no features, no script tags Tai Le ✅ *Tai Tham ❌ Tai Viet ✅ Takri ✅ Tibetan ✅ Tifinagh ❌ Tirhuta ✅ Wancho ✅ Yezidi - no font Zanabazar Square ❌ because of problems with categories and USE shaping implementation. @dscorbett does the same as explained here: https://github.com/dscorbett/zanashinar#script-and-language

[davelab6]

I propose we keep this issue open until all are green checks or there is a separate issue that describes what the script specific situation is and how to fix it

[BrianKraimerMT]

Hello David,

Kamal will respond to this shortly.

[davelab6]

@behdad Noto is actively maintained, so saying no one has bandwidth isn't really fair. Perhaps that was true in the intervening past few years.

As to the accusation that Monotype didn't deliver, I have to say that their invoices were all approved and they remain in good standing with Google. Zachary has given the receipts, so I'd like to kindly suggest that you apologise.

Generally, type designer firms and folks are service providers, so if the client hasn't specified up front what their requirements are, then a positive atmosphere is required to collaborate on negotiating what is and what is not within the scope of an agreed service provision engagement. When working at the forefront of technology, it can be impossible to specify requirements up front.

USE was very fresh at that time, so I think the situation today is fine.

[kmansourMT]

I was about to add that USE was not in the picture at the time the Noto work began. This applies in particular to Tai Tham which still presents some ambiguities to this day, USE or not. As for Elymaic, it is an off-shoot of Aramaic script. Make it conform to USE make as little sense as doing so for Hebrew script. Tifinagh has no features that require USE since it is a simple script (comparable to Latin) that requires a fair number of combining marks.

Finally, we were never contractually required to comply with USE, but we did so anyway as it became possible to do so for fonts with Indic features.

Kamal

[tiroj]

As for Elymaic, it is an off-shoot of Aramaic script. Make it conform to USE make as little sense as doing so for Hebrew script. Tifinagh has no features that require USE since it is a simple script (comparable to Latin) that requires a fair number of combining marks.

Windows 10 passes Tifinagh to USE. Not sure what it does with Elymaic, but it would make some sense to pass it to the Hebrew shaping engine (as is done with Thaana on the grounds that it happens to use the same layout model despite being completely unrelated to Hebrew).

While USE provides some specific modelling of shaping for complex scripts, I think it is a mistake to assume that non-complex scripts should not be passed to USE in the absence of another explicitly identified shaping engine route. In addition to its complex script shaping models, USE provides access to OTL features generally—including standard typographic and discretionary features — in a way that is more consistent than in other shaping engines. While it is obviously up to platform developers to determine how to route specific runs after itemisation, my own inclination would be to pass any script tag that isn't already linked to a specific shaping engine via USE, as this provides the most complete and flexible support for OTL features that may occur in any font. It is, after all, called the Universal Shaping Engine.

[NorbertLindenberg]

Elymaic is listed in the documentation of the Universal Shaping Engine: https://docs.microsoft.com/en-us/typography/script-development/use The USE covers several other right-to-left scripts.

[kmansourMT]

If a script uses only common (script-independent) features like ccmp or liga, how can it be non-compliant with USE?

[NorbertLindenberg]

First, ccmp and liga are not script-independent: They are not documented for all shaping engines, and while HarfBuzz applies them across scripts (at least for horizontal text), Windows doesn’t.

Second, possible issues when moving from the simple shaping engine to USE:

• The USE applies a cluster model based on Unicode character data and numerous overrides, and inserts dotted circles where incoming text doesn’t match that model. The simple shaping engine doesn’t do that.

• The USE applies features in the basic cluster formation phase on a per-cluster basis; features in later phases on a per-run basis. ccmp is applied in the basic cluster formation phase, so lookups can’t see beyond cluster boundaries. For the simple shaping engine, this is not specified, but implementations seem to apply lookups to the entire run. As a result, some lookups that work fine in the simple shaping engine may not work in the USE.

[dscorbett]

There is no need to keep this issue open. The fonts for Elymaic, Tifinagh, and Zanabazar Square work fine with the default shaper. USE supports those scripts, but that doesn’t mean font designers have to use it; it’s just an option if they want to. Switching now has no benefit and could introduce subtle bugs. Noto Sans Tai Tham is broken, but moving to USE would probably make fixing it harder, since USE doesn’t really support Tai Tham.

[davelab6]

There is no need to keep this issue open. The fonts for Elymaic, Tifinagh, and Zanabazar Square work fine with the default shaper. USE supports those scripts, but that doesn’t mean font designers have to use it; it’s just an option if they want to. Switching now has no benefit and could introduce subtle bugs.

Alright, sounds good. It seems like this ought to go into future detailed Noto docs (cc @twardoch)

Noto Sans Tai Tham is broken, but moving to USE would probably make fixing it harder, since USE doesn’t really support Tai Tham.

What's the status of fixing USE to properly support Tai Tham? Currently no momentum and at a dead stop? Going to be fixed this year? :)

[roozbehp]

What's the status of fixing USE to properly support Tai Tham? Currently no momentum and at a dead stop? Going to be fixed this year? :)

Pending action by Unicode. We last met specifically about it in late 2018 to discuss. Report here: https://www.unicode.org/L2/L2018/18332-tai-tham-ad-hoc-report.pdf

One of our key contributors, Martin Hosken, who is our key contact on the ground, has been blocked by his work on Western Cham. We are mostly waiting on him, as well for others to finish their action items.

[davelab6]

Thanks roozbeh!

[davelab6]

@nizarsq asked

suggesting on how I should create test case?

@dscorbett and all, do you have ideas for such test cases? :)

[dscorbett]

This issue is not a bug report: it’s a reminder to check that all Noto fonts that use USE do so properly. As such, it won’t have a test case like other issues, which I assume is what @nizarsq was asking about. Test cases can be created for specific incompatibilities if and when they are discovered.

[marekjez86]

@punchcutter @NorbertLindenberg @davelab6 @roozbehp @dscorbett @tiroj @kmansourMT @nizarsq : I wanted to thank you all for the contribution to this issue. I'll use the table created by @punchcutter to either create individual script issues or as a reminder to check a new script (or set requirements). Closing this issue.