Lots of regressions

[simoncozens]

I broke ra subscript: భ్ద్ర, క్త్రో

Was: Now:

I broke kerning: కుష్నే

Was: Now:

I broke spacing: నిల్వౌం

Was: Now:

I'm sorry, this is a complete mess. We now have a much better QA system which will help us to catch this kind of thing, but I need to deploy it automatically for Noto builds.

[chrissimpkins]

Are any of the glyphs in this string involved?

కనెక్ట్ చేసిన పరికరం

[simoncozens]

I don't think so; looks fine to me:

[simoncozens]

2.001 (MTI source) does not have this issue (not your issue); 2.002 (Fea conversion) does have it.

[chrissimpkins]

We may need to address the third glyph in the UI builds. This is from font files that the Android team shared with me. Includes NS Telugu UI v2.003 below:

[chrissimpkins]

cc @m4rc1e

Do we understand why our diffenator font regression testing did not pick this up in the non-UI family?

[simoncozens]

Your Android issue is due to the UI VF build process. Here's the full story:

• Originally, we had UI and non-UI statics. The UI and non-UI builds for several of the Noto fonts shared the same sources (and hence the same masters) but differed in the way that statics were exported. The UI statics were generated by applying various transformations (dropping the baseline, scaling, feature replacement, glyph swapping, etc.) using Glyphs custom parameters on each instance definitions.
• Then we moved to VFs. The way that we build VFs in fontmake operates only on masters, and of course the UI-specific transformations were not applied to the masters, only to the instances. So when we generated "UI" VFs, we later discovered that they didn't actually have any of the UI transformations applied; they were identical to the non-UI VFs. Hence, when we moved to VFs, we lost the ability to generate UI builds.
• To mitigate this, Marek had me write a VF hotfixer which looked at the Glyphs sources and the VF binary, and manually applied the custom parameters to the binary; replacing the glyphs for UI versions in the binary is a horrible process involving traversing the GSUB and GPOS tables and manipulating them to point at the alternate glyphs.

It's this hotfix process which is causing the problem which you are seeing. Specifically:

• In both cases, we have a buffer katelu=0|viramatelu=0|ttatelu=0|viramatelu=0
• In the non-UI case, the bwlf feature ligates virama/tta to a subscript form: katelu=0|ttasubscripttelu=0|viramatelu=0
• In the UI case, the bwlf features does the same but our VF hotfixer has (correctly) swapped out the ttasubscripttelu in the substitution for a ttasubscriptUItelu. All good so far.
• Next, the haln feature applies to the first katelu and final viramatelu skipping the marks in between. The marks to skip are stored in a mark attachment class definition in the GDEF table. This definition includes ttasubscripttelu, so in the non-UI case, this matches and katelu|viramatelu is substituted for a halant form kahalanttelu.
• The hotfix script did not modify the GDEF table (oops), so when the shaper sees katelu|ttasubscriptUItelu| viramatelu the ttasubscriptUItelu glyph is not skipped over, and the substitution does not match.

The proximate fix here is to make the UI VF hotfix script also do its dirty magic to the GDEF table.

I hope this goes some way to explaining why I really, really don't like touching the UI builds...

[m4rc1e]

@chrissimpkins I think the UI fonts predate diffenator2. I also don't see any UI prs in google/fonts itself, https://github.com/google/fonts/pulls?q=is%3Apr+telugu+is%3Aclosed

[chrissimpkins]

Thanks! My question was about the non-UI builds. My understanding is that this issue refers to the non-UI family and that we pushed v2.003 to prod with these regressions via a PR on the google/fonts repository where we have had diffenator in place.

[m4rc1e]

My understanding is that this issue refers to the non-UI family and that we pushed v2.003 to prod with these regressions via a PR on the google/fonts repository where we have had diffenator in place.

AFAIK, the v2.003 Telugu was merged 6th July 2022. First commit of diffenator2 was 22nd Aug 2022. I reckon I must have reviewed it with the original diffenator which was atrocious at picking up these shaping issues.

[chrissimpkins]

Copy that. Thanks!

[chrissimpkins]

In a report to the Android team

[simoncozens]

Okay. The first text is the word సెర్చ్ which is broken both in UI and non-UI builds. This is rather embarrassing for Google as the text is the word "search". The virama should be applied to the ra, skipping over the subjoiner.

The next word is నెట్వర్క్ "network", same issue with the ra and also the wa should subjoin the tta. (I can't reproduce that failure, however.)

The next issue another mark filtering set problem, word is "connect" కనెక్ట్ but ka + any subjoiner + virama will show it. The next text is the same problem but with a ర instead of a క. Both of these are only problems in the UI build; non-UI font is fine.

The final text, I can't quite read, but it looks like there are a couple of issues; subjoined ra not working as mentioned above, and another virama attaching to a post base instead of the consonant (mark filtering set issue).

[chrissimpkins]

Some of these look like they could be duplicates but continuing to share other images that are coming up in new issue trackers so that you have the entire picture of reported issues.

[chrissimpkins]

[simoncozens]

Some updates:

• I fixed subscript ra (భ్ద్ర); two lookups had been combined into one, and this was causing some substitutions not to fire.
• Halant forms not forming with subscripts (సెర్చ్) - and possibly many of the more recent (nonUI) issues: this was a glyphsLib bug! Oddly, I already fixed the bug in a completely unrelated context, but that hasn't made it into a glyphsLib release yet. So when glyphsLib is released, most of this issue will magically get fixed.
• The kerning is difficult one. I'm not sure if it's even possible to fix this in our current fontmake workflow. This is because ufo2ft kerns bases against bases ignoring marks, but we make use of spacing marks. In "కుషే" "కషే" we want to insert a kern between the second pair (కషే ) to stop the two above marks banging into each other. But when we have "కుషే" we don't need a kern. And if the vowel sign "ు" was a base, we wouldn't get a kern; but because it's a mark, it gets skipped over and the kern is applied.
• The "spacing" issue is the same thing. We probably need a way to tell the kern feature writer not to skip marks, since we are using spacing mark glyphs so heavily as we make layout decisions.

[simoncozens]

• The second part of the ra subscript problem (క్త్రో) is the same bug as #12, so I'm going to ignore it here and fix it over there.
• I fixed the kerning issue in ufo2ft, but it will require merging and a new release and so on.
• That leaves the UI issues (hats of క and ర not getting cut off when virama applied to post-base conjunct forms). I'm going to open that in a new issue, and then everything here is resolved.

(Modulo glyphsLib/ufo2ft updates.)

[simoncozens]

Note to self: if ra-subscript starts failing inexplicably, before fiddling with the feature code (which is fine) ensure you have the latest glyphsLib and ufo2ft.