Set overlap bit on every glyph in every VF and keep it set

[davelab6]

In order for TT VFs to be correctly rendered with outlines, each glyph in the glyf table needs to have the overlap bit set, or ideally each glyph that has overlaps although that's more complicated to implement.

And we need string regression checks to prevent that slipping, and that will be easier as well if it's just all VF glyphs I suppose.

[aaronbell]

Don’t know if it is useful but this is the code I put in Cascadia Code’s build script to solve this problem:

def set_overlap_flag(varfont: fontTools.ttLib.TTFont) -> fontTools.ttLib.TTFont:
    glyf = cast(_g_l_y_f.table__g_l_y_f, varfont["glyf"])
    for glyph_name in glyf.keys():
        glyph = glyf[glyph_name]
        if glyph.isComposite():
            # Set OVERLAP_COMPOUND bit for compound glyphs
            glyph.components[0].flags |= 0x400
        elif glyph.numberOfContours > 0:
            # Set OVERLAP_SIMPLE bit for simple glyphs
            glyph.flags[0] |= 0x40

[madig]

I remember reading on the FreeType devel list that FT will switch to more expensive supersamling mode when encountering overlap bits, which is why setting them indiscriminately may be a bad idea. The script by @aaronbell is what we use in production to quickly set the bit if needed, per glyph :) No idea if something needs to be done for CFF2.

[chrissimpkins]

I remember reading on the FreeType devel list that FT will switch to more expensive supersamling mode when encountering overlap bits, which is why setting them indiscriminately may be a bad idea. The script by @aaronbell is what we use in production to quickly set the bit if needed, per glyph :) No idea if something needs to be done for CFF2.

This thread summarizes the state of the conversation around this as of late 2020 https://github.com/fonttools/fonttools/issues/2059

[garretrieger]

Don’t know if it is useful but this is the code I put in Cascadia Code’s build script to solve this problem:

def set_overlap_flag(varfont: fontTools.ttLib.TTFont) -> fontTools.ttLib.TTFont:
    glyf = cast(_g_l_y_f.table__g_l_y_f, varfont["glyf"])
    for glyph_name in glyf.keys():
        glyph = glyf[glyph_name]
        if glyph.isComposite():
            # Set OVERLAP_COMPOUND bit for compound glyphs
            glyph.components[0].flags |= 0x400
        elif glyph.numberOfContours > 0:
            # Set OVERLAP_SIMPLE bit for simple glyphs
            glyph.flags[0] |= 0x40

FYI fonttools has pretty much this exact function in the instancer library: https://github.com/fonttools/fonttools/blob/main/Lib/fontTools/varLib/instancer/__init__.py#L1075

[davelab6]

https://jsbin.com/xewuzew has examples of the problems which can better solved when these bits are set

[twardoch]

Please also keep in mind that these flags don’t currently make it into WOFF2 (there is no space for them). There is a proposal for revision to include them ( https://www.w3.org/TR/WOFF2/#p1 ) but I don’t know the implementation status of it. Certainly WOFF2 encoders (like fontTools or woff2_compress) and WOFF2 decoders (the libraries that are used by the browsers) will have to be updated to take this into account.

The bits are retained in WOFF (1.0).

[twardoch]

I don’t know when these changes into WOFF2 will be adopted (into the spec, and into the software). Or perhaps something already silently is supporting these changes? If not, then for now I believe still WOFF 1.0 is a "safer" format for VF delivery than WOFF2.

[rsheeter]

ideally each glyph that has overlaps although that's more complicated to implement.

I fear that is indeed what we want, unless we think the reality is that most glyphs do have overlaps so we wouldn't really save much by avoiding setting it unnecessarily.

There is a proposal for revision to include them ( https://www.w3.org/TR/WOFF2/#p1 ) but I don’t know the implementation status of it.

@garretrieger when do you expect the code change for this to hit https://github.com/google/woff2?

[dberlow]

Lorp: “It seems to me that the spec has been 100% clear on this since the early 1990s, and therefore I don’t agree with any suggestions to bypass it.”

1989.

Adam: “ Please also keep in mind that these flags don’t currently make it into WOFF2 (there is no space for them).”

wow.

[chrissimpkins]

unless we think the reality is that most glyphs do have overlaps so we wouldn't really save much by avoiding setting it unnecessarily.

A quick glance through one master in a large multi-script project suggests that the ballpark may be many rather than most. n=1.

[madig]

I wrote this script to set the overlap bits and count how many overlaps there are:

from __future__ import annotations

import argparse
from typing import Any, Mapping

import pathops
from fontTools.ttLib import TTFont
from fontTools.ttLib.removeOverlaps import componentsOverlap, skPathFromGlyph
from fontTools.ttLib.tables import _g_l_y_f

def set_overlap_bits_if_overlapping(varfont: TTFont) -> tuple[int, int]:
    glyph_set = varfont.getGlyphSet()
    glyf_table: _g_l_y_f.table__g_l_y_f = varfont["glyf"]
    flag_overlap_compound = _g_l_y_f.OVERLAP_COMPOUND
    flag_overlap_simple = _g_l_y_f.flagOverlapSimple
    overlapping_contours = 0
    overlapping_components = 0
    for glyph_name in glyf_table.keys():
        glyph = glyf_table[glyph_name]
        # Set OVERLAP_COMPOUND bit for compound glyphs
        if glyph.isComposite() and componentsOverlap(glyph, glyph_set):
            overlapping_components += 1
            glyph.components[0].flags |= flag_overlap_compound
        # Set OVERLAP_SIMPLE bit for simple glyphs
        elif glyph.numberOfContours > 0 and glyph_overlaps(glyph_name, glyph_set):
            overlapping_contours += 1
            glyph.flags[0] |= flag_overlap_simple
    return (overlapping_contours, overlapping_components)

def glyph_overlaps(glyph_name: str, glyph_set: Mapping[str, Any]) -> bool:
    path = skPathFromGlyph(glyph_name, glyph_set)
    path2 = pathops.simplify(path, clockwise=path.clockwise)  # remove overlaps
    if path != path2:
        return True
    return False

parser = argparse.ArgumentParser()
parser.add_argument("font", nargs="+", type=TTFont)
parsed_args = parser.parse_args()
fonts: list[TTFont] = parsed_args.font

for font in fonts:
    ocont, ocomp = set_overlap_bits_if_overlapping(font)
    num_glyphs = font["maxp"].numGlyphs
    ocont_p = ocont / num_glyphs
    ocomp_p = ocomp / num_glyphs
    print(
        font.reader.file.name,
        f"{num_glyphs} glyphs, {ocont} overlapping contours ({ocont_p:.2%}), {ocomp} overlapping components ({ocomp_p:.2%})",
    )
    font.save("a.ttf")

I get the following exemplary results for some git master copies:

> py mark-overlappers.py NotoSans-VF.ttf NotoSansCJKjp-VF.ttf
NotoSans-VF.ttf 3748 glyphs, 535 overlapping contours (14.27%), 44 overlapping components (1.17%)
NotoSansCJKjp-VF.ttf 65535 glyphs, 64169 overlapping contours (97.92%), 0 overlapping components (0.00%)

[chrissimpkins]

Lol. Range 14% - 98% overlapping. Now we know. :)

[madig]

If you want, I can run the script over https://github.com/googlefonts/noto-fonts/tree/main/unhinted/variable-ttf and also try to write a program to benchmark FreeType (not sure how long that would take though and how useful it would be -- I imagine setting the bits on the glyphs that need them and praying it's not too slow is the best we can do).

[madig]

Actually, running it over all of Noto's VFs is easy:

NotoKufiArabic-VF.ttf 850 glyphs, 469 overlapping contours (55.18%), 4 overlapping components (0.47%)
NotoLoopedLao-VF.ttf 181 glyphs, 117 overlapping contours (64.64%), 0 overlapping components (0.00%)
NotoLoopedLaoUI-VF.ttf 151 glyphs, 90 overlapping contours (59.60%), 0 overlapping components (0.00%)
NotoLoopedThai-VF.ttf 212 glyphs, 143 overlapping contours (67.45%), 0 overlapping components (0.00%)
NotoLoopedThaiUI-VF.ttf 157 glyphs, 106 overlapping contours (67.52%), 0 overlapping components (0.00%)
NotoNaskhArabic-VF.ttf 1614 glyphs, 95 overlapping contours (5.89%), 486 overlapping components (30.11%)
NotoNaskhArabicUI-VF.ttf 1614 glyphs, 94 overlapping contours (5.82%), 486 overlapping components (30.11%)
NotoNastaliqUrdu-VF.ttf 1406 glyphs, 254 overlapping contours (18.07%), 0 overlapping components (0.00%)
NotoRashiHebrew-VF.ttf 105 glyphs, 53 overlapping contours (50.48%), 0 overlapping components (0.00%)
NotoSans-Italic-VF.ttf 3762 glyphs, 523 overlapping contours (13.90%), 39 overlapping components (1.04%)
NotoSans-VF.ttf 3748 glyphs, 535 overlapping contours (14.27%), 44 overlapping components (1.17%)
NotoSansAdlam-VF.ttf 361 glyphs, 25 overlapping contours (6.93%), 0 overlapping components (0.00%)
NotoSansAdlamUnjoined-VF.ttf 155 glyphs, 14 overlapping contours (9.03%), 0 overlapping components (0.00%)
NotoSansArabic-VF.ttf 1661 glyphs, 365 overlapping contours (21.97%), 529 overlapping components (31.85%)
NotoSansArabicUI-VF.ttf 1576 glyphs, 466 overlapping contours (29.57%), 451 overlapping components (28.62%)
NotoSansArmenian-VF.ttf 107 glyphs, 52 overlapping contours (48.60%), 0 overlapping components (0.00%)
NotoSansBalinese-VF.ttf 361 glyphs, 234 overlapping contours (64.82%), 14 overlapping components (3.88%)
NotoSansBamum-VF.ttf 662 glyphs, 325 overlapping contours (49.09%), 0 overlapping components (0.00%)
NotoSansBassaVah-VF.ttf 49 glyphs, 1 overlapping contours (2.04%), 0 overlapping components (0.00%)
NotoSansBengali-VF.ttf 695 glyphs, 423 overlapping contours (60.86%), 36 overlapping components (5.18%)
NotoSansBengaliUI-VF.ttf 695 glyphs, 423 overlapping contours (60.86%), 36 overlapping components (5.18%)
NotoSansCanadianAboriginal-VF.ttf 762 glyphs, 177 overlapping contours (23.23%), 0 overlapping components (0.00%)
NotoSansCham-VF.ttf 131 glyphs, 77 overlapping contours (58.78%), 1 overlapping components (0.76%)
NotoSansCherokee-VF.ttf 273 glyphs, 36 overlapping contours (13.19%), 0 overlapping components (0.00%)
NotoSansDevanagari-VF.ttf 993 glyphs, 474 overlapping contours (47.73%), 72 overlapping components (7.25%)
NotoSansDisplay-Italic-VF.ttf 3327 glyphs, 518 overlapping contours (15.57%), 34 overlapping components (1.02%)
NotoSansDisplay-VF.ttf 3744 glyphs, 510 overlapping contours (13.62%), 48 overlapping components (1.28%)
NotoSansEthiopic-VF.ttf 594 glyphs, 521 overlapping contours (87.71%), 0 overlapping components (0.00%)
NotoSansGeorgian-VF.ttf 225 glyphs, 16 overlapping contours (7.11%), 0 overlapping components (0.00%)
NotoSansGujarati-VF.ttf 830 glyphs, 376 overlapping contours (45.30%), 24 overlapping components (2.89%)
NotoSansGunjalaGondi-VF.ttf 254 glyphs, 156 overlapping contours (61.42%), 0 overlapping components (0.00%)
NotoSansGurmukhi-VF.ttf 344 glyphs, 110 overlapping contours (31.98%), 9 overlapping components (2.62%)
NotoSansGurmukhiUI-VF.ttf 344 glyphs, 110 overlapping contours (31.98%), 9 overlapping components (2.62%)
NotoSansHanifiRohingya-VF.ttf 179 glyphs, 33 overlapping contours (18.44%), 67 overlapping components (37.43%)
NotoSansHebrew-VF.ttf 149 glyphs, 29 overlapping contours (19.46%), 0 overlapping components (0.00%)
NotoSansHebrewDroid-VF.ttf 179 glyphs, 24 overlapping contours (13.41%), 0 overlapping components (0.00%)
NotoSansHebrewNew-VF.ttf 149 glyphs, 29 overlapping contours (19.46%), 0 overlapping components (0.00%)
NotoSansKannada-VF.ttf 655 glyphs, 301 overlapping contours (45.95%), 45 overlapping components (6.87%)
NotoSansKannadaUI-VF.ttf 655 glyphs, 301 overlapping contours (45.95%), 45 overlapping components (6.87%)
NotoSansKayahLi-VF.ttf 60 glyphs, 13 overlapping contours (21.67%), 0 overlapping components (0.00%)
NotoSansKhmer-VF.ttf 363 glyphs, 216 overlapping contours (59.50%), 0 overlapping components (0.00%)
NotoSansKhmerUI-VF.ttf 381 glyphs, 233 overlapping contours (61.15%), 0 overlapping components (0.00%)
NotoSansLao-VF.ttf 116 glyphs, 17 overlapping contours (14.66%), 0 overlapping components (0.00%)
NotoSansLaoUI-VF.ttf 118 glyphs, 15 overlapping contours (12.71%), 0 overlapping components (0.00%)
NotoSansLisu-VF.ttf 60 glyphs, 8 overlapping contours (13.33%), 0 overlapping components (0.00%)
NotoSansMalayalam-VF.ttf 365 glyphs, 177 overlapping contours (48.49%), 24 overlapping components (6.58%)
NotoSansMalayalamUI-VF.ttf 364 glyphs, 177 overlapping contours (48.63%), 24 overlapping components (6.59%)
NotoSansMedefaidrin-VF.ttf 97 glyphs, 86 overlapping contours (88.66%), 0 overlapping components (0.00%)
NotoSansMeeteiMayek-VF.ttf 92 glyphs, 70 overlapping contours (76.09%), 0 overlapping components (0.00%)
NotoSansMono-VF.ttf 3787 glyphs, 761 overlapping contours (20.10%), 65 overlapping components (1.72%)
NotoSansMyanmar-VF.ttf 619 glyphs, 355 overlapping contours (57.35%), 10 overlapping components (1.62%)
NotoSansMyanmarUI-VF.ttf 615 glyphs, 370 overlapping contours (60.16%), 1 overlapping components (0.16%)
NotoSansOlChiki-VF.ttf 55 glyphs, 33 overlapping contours (60.00%), 0 overlapping components (0.00%)
NotoSansOriya-VF.ttf 513 glyphs, 359 overlapping contours (69.98%), 33 overlapping components (6.43%)
NotoSansOriyaUI-VF.ttf 513 glyphs, 359 overlapping contours (69.98%), 33 overlapping components (6.43%)
NotoSansSinhala-VF.ttf 645 glyphs, 479 overlapping contours (74.26%), 28 overlapping components (4.34%)
NotoSansSinhalaUI-VF.ttf 645 glyphs, 479 overlapping contours (74.26%), 28 overlapping components (4.34%)
NotoSansSoraSompeng-VF.ttf 42 glyphs, 29 overlapping contours (69.05%), 0 overlapping components (0.00%)
NotoSansSundanese-VF.ttf 89 glyphs, 12 overlapping contours (13.48%), 0 overlapping components (0.00%)
NotoSansSymbols-VF.ttf 1224 glyphs, 500 overlapping contours (40.85%), 63 overlapping components (5.15%)
NotoSansTaiTham-VF.ttf 824 glyphs, 115 overlapping contours (13.96%), 0 overlapping components (0.00%)
NotoSansTamil-VF.ttf 244 glyphs, 112 overlapping contours (45.90%), 14 overlapping components (5.74%)
NotoSansTamilUI-VF.ttf 244 glyphs, 112 overlapping contours (45.90%), 14 overlapping components (5.74%)
NotoSansTangsa-VF.ttf 94 glyphs, 88 overlapping contours (93.62%), 0 overlapping components (0.00%)
NotoSansTelugu-VF.ttf 958 glyphs, 418 overlapping contours (43.63%), 253 overlapping components (26.41%)
NotoSansTeluguUI-VF.ttf 958 glyphs, 418 overlapping contours (43.63%), 253 overlapping components (26.41%)
NotoSansThaana-VF.ttf 90 glyphs, 23 overlapping contours (25.56%), 0 overlapping components (0.00%)
NotoSansThai-VF.ttf 140 glyphs, 46 overlapping contours (32.86%), 0 overlapping components (0.00%)
NotoSansThaiUI-VF.ttf 140 glyphs, 47 overlapping contours (33.57%), 0 overlapping components (0.00%)
NotoSansVithkuqi-VF.ttf 103 glyphs, 61 overlapping contours (59.22%), 0 overlapping components (0.00%)
NotoSerif-Italic-VF.ttf 3703 glyphs, 520 overlapping contours (14.04%), 34 overlapping components (0.92%)
NotoSerif-VF.ttf 3691 glyphs, 533 overlapping contours (14.44%), 29 overlapping components (0.79%)
NotoSerifArmenian-VF.ttf 107 glyphs, 61 overlapping contours (57.01%), 0 overlapping components (0.00%)
NotoSerifBengali-VF.ttf 640 glyphs, 455 overlapping contours (71.09%), 4 overlapping components (0.62%)
NotoSerifDevanagari-VF.ttf 871 glyphs, 396 overlapping contours (45.46%), 32 overlapping components (3.67%)
NotoSerifDisplay-Italic-VF.ttf 3707 glyphs, 722 overlapping contours (19.48%), 31 overlapping components (0.84%)
NotoSerifDisplay-VF.ttf 3268 glyphs, 673 overlapping contours (20.59%), 19 overlapping components (0.58%)
NotoSerifDogra-VF.ttf 143 glyphs, 95 overlapping contours (66.43%), 0 overlapping components (0.00%)
NotoSerifEthiopic-VF.ttf 594 glyphs, 531 overlapping contours (89.39%), 0 overlapping components (0.00%)
NotoSerifGeorgian-VF.ttf 225 glyphs, 13 overlapping contours (5.78%), 0 overlapping components (0.00%)
NotoSerifGujarati-VF.ttf 456 glyphs, 296 overlapping contours (64.91%), 0 overlapping components (0.00%)
NotoSerifGurmukhi-VF.ttf 294 glyphs, 39 overlapping contours (13.27%), 4 overlapping components (1.36%)
NotoSerifHebrew-VF.ttf 152 glyphs, 34 overlapping contours (22.37%), 0 overlapping components (0.00%)
NotoSerifKannada-VF.ttf 417 glyphs, 279 overlapping contours (66.91%), 44 overlapping components (10.55%)
NotoSerifKhmer-VF.ttf 361 glyphs, 258 overlapping contours (71.47%), 0 overlapping components (0.00%)
NotoSerifKhojki-VF.ttf 421 glyphs, 278 overlapping contours (66.03%), 6 overlapping components (1.43%)
NotoSerifLao-VF.ttf 117 glyphs, 57 overlapping contours (48.72%), 0 overlapping components (0.00%)
NotoSerifMalayalam-VF.ttf 355 glyphs, 175 overlapping contours (49.30%), 1 overlapping components (0.28%)
NotoSerifNyiakengPuachueHmong-VF.ttf 76 glyphs, 19 overlapping contours (25.00%), 0 overlapping components (0.00%)
NotoSerifOriya-VF.ttf 690 glyphs, 286 overlapping contours (41.45%), 14 overlapping components (2.03%)
NotoSerifSinhala-VF.ttf 645 glyphs, 470 overlapping contours (72.87%), 25 overlapping components (3.88%)
NotoSerifTamil-VF.ttf 222 glyphs, 132 overlapping contours (59.46%), 0 overlapping components (0.00%)
NotoSerifTamilSlanted-VF.ttf 222 glyphs, 130 overlapping contours (58.56%), 0 overlapping components (0.00%)
NotoSerifTelugu-VF.ttf 728 glyphs, 536 overlapping contours (73.63%), 20 overlapping components (2.75%)
NotoSerifThai-VF.ttf 140 glyphs, 60 overlapping contours (42.86%), 0 overlapping components (0.00%)
NotoSerifTibetan-VF.ttf 1894 glyphs, 1817 overlapping contours (95.93%), 0 overlapping components (0.00%)
NotoSerifToto-VF.ttf 40 glyphs, 21 overlapping contours (52.50%), 0 overlapping components (0.00%)
NotoSerifVithkuqi-VF.ttf 103 glyphs, 66 overlapping contours (64.08%), 0 overlapping components (0.00%)
NotoSerifYezidi-VF.ttf 56 glyphs, 40 overlapping contours (71.43%), 0 overlapping components (0.00%)

So yeah, a lot of variation depending on script I suppose. Running the script was fairly quick though, so I imagine in the typical non-CJK case, checking for overlap is fast enough to be on by default.

[rsheeter]

I wrote this script to set the overlap bits and count how many overlaps there are

Only at the default position in design space iiuc? How likely do we think it is the answer changes as you meander about the designspace?

[rsheeter]

also try to write a program to benchmark FreeType

This would be very informative

[garretrieger]

ideally each glyph that has overlaps although that's more complicated to implement.

I fear that is indeed what we want, unless we think the reality is that most glyphs do have overlaps so we wouldn't really save much by avoiding setting it unnecessarily.

There is a proposal for revision to include them ( https://www.w3.org/TR/WOFF2/#p1 ) but I don’t know the implementation status of it.

@garretrieger when do you expect the code change for this to hit https://github.com/google/woff2?

I have an updated implementation for https://github.com/google/woff2 written up that I'm just finalizing. Hopefully should be able to PR it next week.

[madig]

Only at the default position in design space iiuc? How likely do we think it is the answer changes as you meander about the designspace?

Yes :) Hm! I have no idea and it would take me some time to test this.

[dberlow]

The main thing is as you say, the identifying and flagging the default.

As far as I know, in fonts we’ve done, no contour will meander from overlap to non-overlap. But as weight increases greatly, there are a very few cases where the contours go from non-overlap to overlap.

On Mar 25, 2022, at 1:45 PM, rsheeter @.***> wrote:

 I wrote this script to set the overlap bits and count how many overlaps there are

Only at the default position in design space iiuc? How likely do we think it is the answer changes as you meander about the designspace?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.

[rsheeter]

I think perhaps checking at a set of positions (I'd probably use the fallbacks from our axis registry + min/max of each axis as a starting point) in designspace might be worthwhile then, otherwise missing overlaps that appear at high weight seems all too likely.

[m4rc1e]

Here's a small back of the napkin FT benchmark I've cobbled together for Open Sans VF. It renders each glyph which contains overlaps across a range of different sizes.

import sys
import freetype
import time

glyphs = ['%', ':', '=', 'B', '¦', '©', '«', '®', '±', '¼', '½', '¾', 'Å', 'Ç', 'ç', 'Ą', 'ą', 'Ę', 'ę', 'Į', 'į', 'Ş', 'ş', 'ţ', 'ŧ', 'Ų', 'ų', 'Ǫ', 'ǫ', 'Ǭ', 'ǭ', 'ǻ', '˝', '̏', '΅', 'Θ', 'Ξ', 'Σ', 'Φ', 'ѧ', 'Ѽ', 'ѽ', '҈', '҉', 'Ҋ', 'Ҙ', 'ҙ', 'Ҝ', 'ҝ', 'Ҟ', 'Ҫ', 'ҫ', 'Ӄ', 'Ӻ', 'ְ', 'ֱ', '῞', '‰', '″', '⁼', '₧', '₪', '℅', '№', '⅛', '⅜', '⅝', '⅞']

if len(sys.argv) != 2:
    print("Usage: python render_test.py font.ttf")
    sys.exit()

font_fp = sys.argv[1]

face = freetype.Face(font_fp)

start = time.time()
for size in range(8, 300):
    for c in glyphs:
        face.set_char_size(size*64)
        face.load_char(c)
        bitmap = face.glyph.bitmap.buffer
print(f"finished in {time.time() - start}")

Open Sans without the bits set (in ofl/opensans) : 20.3 secs With bits set using Niko's script: 24.25 secs

That's quite a spicy ~20% difference. More than happy to dynamically get fonts overlaps if people want to test this on other families.

[madig]

Here's a new script that checks all fvar instance locations. It returns higher percentages for some fonts I tested it on.

NotoSans-VF.ttf 3748 glyphs, 1651 overlapping contours (44.05%), 1444 overlapping components (38.53%)

from __future__ import annotations

import argparse
from pathlib import Path

import pathops
import uharfbuzz as hb
from fontTools.ttLib import TTFont
from fontTools.ttLib.tables import _g_l_y_f

def overlapping_glyphs(
    font: hb.Font, coordinates: list[dict[str, float]], num_glyphs: int
) -> set[int]:
    """Return the set of glyph IDs that overlap in any of the user
    coordinates."""
    overlapping = set()
    for gid in range(num_glyphs):
        for coordinate in coordinates:
            font.set_variations(coordinate)
            path = pathops.Path()
            font.draw_glyph_with_pen(gid, path.getPen())
            path2 = pathops.simplify(path, clockwise=path.clockwise)  # Remove overlaps.
            if path != path2:
                overlapping.add(gid)
                break  # If any coordinate has the glyph overlapping, the bit must be set for all.
    return overlapping

def set_overlap_bits_if_overlapping(
    varfont: TTFont, overlapping_glyphs: set[int]
) -> tuple[int, int]:
    name_mapping = varfont.getGlyphNameMany(overlapping_glyphs)
    glyf_table: _g_l_y_f.table__g_l_y_f = varfont["glyf"]

    overlapping_contours = 0
    overlapping_components = 0
    for glyph_name in name_mapping:
        glyph = glyf_table[glyph_name]
        # Set OVERLAP_COMPOUND bit for compound glyphs
        if glyph.isComposite():
            overlapping_components += 1
            glyph.components[0].flags |= _g_l_y_f.OVERLAP_COMPOUND
        # Set OVERLAP_SIMPLE bit for simple glyphs
        elif glyph.numberOfContours > 0:
            overlapping_contours += 1
            glyph.flags[0] |= _g_l_y_f.flagOverlapSimple

    return (overlapping_contours, overlapping_components)

parser = argparse.ArgumentParser()
parser.add_argument("font", nargs="+", type=Path)
parsed_args = parser.parse_args()
fonts: list[Path] = parsed_args.font

for font_path in fonts:
    font = TTFont(font_path)
    num_glyphs = font["maxp"].numGlyphs
    fvar = font["fvar"]

    instance_coordinates = [instance.coordinates for instance in fvar.instances]
    hbfont = hb.Font(hb.Face(hb.Blob.from_file_path(font_path)))
    overlapping_glyphs = overlapping_glyphs(hbfont, instance_coordinates, num_glyphs)

    ocont, ocomp = set_overlap_bits_if_overlapping(font, overlapping_glyphs)
    ocont_p = ocont / num_glyphs
    ocomp_p = ocomp / num_glyphs
    print(
        font.reader.file.name,
        f"{num_glyphs} glyphs, "
        f"{ocont} overlapping contours ({ocont_p:.2%}), "
        f"{ocomp} overlapping components ({ocomp_p:.2%})",
    )
    font.save("a.ttf")

[davelab6]

Sadly fvar instances in most gf families are highly limited to the weight axis only, or weight and width. STAT table instances will be more comprehensive.

[anthrotype]

I think it'd be better to compare the signed area of the two glyphs' paths before and after overlap removal, instead of comparing them for equality, which may be too sensible to false positive. pathops.Path has an area property that returns that. If the signed area changed (within some tolerance) then some overlaps have been removed.

[skef]

@madig I recommend looking at some particular glyphs run against your script, as pathops.simplify() does more than just remove overlap. For example it appears to merge colinear segments. (And don't forget inflection points, although I guess those aren't an issue for TT renderers.)

The extra cleanup may not sound central to the problem but glyphs designed for compatibility of interpolation will often have "cleanable" features in one instance that aren't so in another. I've been experimenting with another (mostly Latin) font and got similar numbers to you based on path comparison but the percentage of glyphs with actual overlap (as measured by detecting segment intersections at 0 < t < 1) was more like 20%.

[skef]

If folks can confirm that glyph "overlap" <=> glyph segment intersections I'd be more likely to attempt a quickish pass at https://github.com/fonttools/skia-pathops/issues/55 . (The scare-quotes because the category also includes self-intersecting paths such as the little triangle you see in "K"s and inverted curves between stroke offset cusps.)

[madig]

You mean check for intersections of any segment of a contour with any other and then setting the bits? Will try when I get around to it.

[skef]

@madig yes, I think that should cover 99.9% of "overlap" cases. There can still be contours with non-intersecting embedded contours of the same direction but its unlikely in production fonts.

[m4rc1e]

I've modified Niko's script so it compares path areas and the results are much better. If I run Niko's original script on Open Sans VF, I get ~90 glyphs with overlaps, most of these are false positives. If I use the modified script which compares areas, I get 27 which is much better.

As @skef pointed out, pathops.simplify is doing more than just removing overlaps. It will sometimes move/add points by a small amount etc. To take these modifications into consideration, I rounded the areas to integers and then did the comparison. This removes further false positives

I've been thinking about how to implement this into the tool chain and I'm thinking of doing the following:

• Move Niko's modified overlap_glyph function to https://github.com/fonttools/fonttools/blob/main/Lib/fontTools/ttLib/removeOverlaps.py
• When ufo2ft is generating the static fonts for the VF, we will check and set each glyph which has overlaps
• When varLib??? is generating the VF, we will iterate over each glyph and if any glyph has the bit enabled in the static fonts, we'll enable it for the VF.

The above should be able to solve https://github.com/google/fonts/issues/4405#issuecomment-1080774855

[anthrotype]

by "static fonts" I suppose you mean the master fonts here. It's not sufficient that we set the bit in the VF if any of the masters have the bit set; it may well be that glyphs don't overlap in any of the masters but they do overlap somewhere in the runtime-interpolated instances; and not necessarily one of the named fvar instances either (see @davelab6 comment above where he suggests using the combinations of the STAT axis values)

[m4rc1e]

by "static fonts" I suppose you mean the master fonts here

Correct

it may well be that glyphs don't overlap in any of the masters but they do overlap somewhere in the runtime-interpolated instances

Do we have an example of this? I don't think I've seen it.

[anthrotype]

Do we have an example of this?

in theory it is possible, even though you may argue not very frequent in real world fonts. Imagine two circles that swap their respective position as they get interpolated from one master to another; they would overlap in between when they cross paths.

[m4rc1e]

Imagine two circles that swap their respective position as they get interpolated from one master to another

Just had the same thought :-)

I don't think checking the STAT axis values is going to do much good either. You're banking on the dev declaring an axis value for the collision which may also not be the case.

[anthrotype]

We probably need a way for a font developer to specify that particular glyphs need to have the overlap flag set, since they already know this information, by storing that in the sources the compiler can skip the relative expensive operation. If that's missing, we can do a rough check across all the masters + named instances, or combinations of STAT axis values if you like. I propose we define something like a public.glyphOverlaps or similar, containing a list of glyph names, and store that in the designspace <lib> element. Ufo2ft would check that and set the flags accordingly if present, otherwise fall back to the proposed skia-pathops-based heuristic.

[dberlow]

I have examples of what I call “waxing” and “waning” overlaps. The former being overlaps do not exist in a master but do occur as in the circles example above, and overlaps that exist in masters, but disappear in others, as would be the situation with the variation from an overlapping y-stem opening a gap for a stencil style.

I am almost certain the only way to flag everything for overlaps is looking for then in each of the design space’s masters and in each 50% interpolation between each master.

[m4rc1e]

I am almost certain the only way to flag everything for overlaps is looking for then in each of the design space’s masters and in each 50% interpolation between each master.

I've dabbled with this idea by using a product. Unfortunately, we end up with runtime of 3^N where N is the number of axes and 3 is min,mid and max for each axis. Unfortunately for mega families such as Roboto Flex which have 13 axes, it means we need to check 1594323 combos which is unfeasible!

[dberlow]

I was not suggesting anything runtime of this length, but rather I thought we were talking about us, the developers, placing the flags on glyphs with overlaps.

If that is the case, we can flag for two things:

1. the glyphs that have overlap anywhere in the design space.
2. the specific ranges of the design space with overlaps of each glyph.

On Wed, Apr 13, 2022 at 8:34 AM Marc Foley @.***> wrote:

I am almost certain the only way to flag everything for overlaps is looking for then in each of the design space’s masters and in each 50% interpolation between each master.

I've dabbled with this idea by using a product. Unfortunately, we end up with runtime of 3^N where N is the number of axes and 3 is min,mid and max for each axis. Unfortunately for mega families such as Roboto Flex which have 13 axes, it means we need to check 1594323 combos which is unfeasible!

— Reply to this email directly, view it on GitHub https://github.com/google/fonts/issues/4405#issuecomment-1097997234, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAO5VDTARLD7OZ6PTS2N4G3VE25L3ANCNFSM5RRE56YQ . You are receiving this because you commented.Message ID: @.***>

[m4rc1e]

1. the glyphs that have overlap anywhere in the design space.

If you search the designspace, you'll end up with a product like I've mentioned above. I don't know how you can avoid it.

[skef]

@m4rc1e, if one desired going down this road, couldn't one start by picking a representative instance (such as the default) and checking the contours of each other instance for left-right or up-down contour order inversions (probably by sub-path bounding box)? That way you a) cut off the need to do further analysis when there are no such inversions and b) can greatly reduce the number of intermediate contour checks needed in all but the strangest cases.

[m4rc1e]

@skef I like this idea but due to its complexity, I'm wondering if it's worth it. How bad will it be if we just enable these bits for all glyphs? will budget Android phones grind to a halt? will users even notice? I made a back of the napkin test in https://github.com/google/fonts/issues/4405#issuecomment-1080541758 but this doesn't really indicate real world usage,.

@bungeman apologies for tagging you again but I'm wondering how you'd approach this problem and what the repercussions will be for the above questions as well? I'm struggling to come up with a decent heuristic which will allow us to enable the glyph bits only for overlapping glyphs. I've tried exhaustively searching a font's whole designspace for overlaps, but this is too slow for fonts which have many axes.

[twardoch]

What if we reverse the logic: we assume to set it everywhere but then we think of optimized ways to CLEAR the bit in safe cases? There sure will be some way to exclude possibility of overlaps in some cases quickly?

[twardoch]

in short: in doubt we set it, we only unset it in cases where it’s easily determined :)

[m4rc1e]

@twardoch It's fairly trivial for us to discover which glyphs have overlapping contours if they're constructed from a single contour. We just need to check each master for overlaps. For glyphs constructed from multiple contours, this is harder to check so perhaps we could just enable the bits for these glyphs?

RobotoFlex has 928 glyphs. 198 of them have 2 or more paths. It doesn't sound like a deal breaker to enable the bits for these since its 21% of the total glyphs.

As far as compromises go, I feel this may work.

[skef]

It's fairly trivial for us to discover which glyphs have overlapping contours if they're constructed from a single contour. We just need to check each master for overlaps.

This will probably be true in practice but it doesn't just "fall out" of the geometry.

One problem I've just recently run across was that a typical four on-curve point dot got mapped wrong in one master, so that the the first point was on the opposite side. The effect is that the dot disappears at some positions in the design space. "Disappearing" amounts to "all of the points were on top of one another."

If you have a more irregular shape mapped in an analogous way you can have no self-intersections any master but self-intersections in some parts of design space. Seeing this in actual use would be very unexpected, of course, especially given that self-intersections are basically never supposed to be "bare", but the geometry itself doesn't prevent it.

[madig]

Copy-pasting my discussion with Cosimo here:

madigens: also, when you get to it, do you have an opinion on https://github.com/google/fonts/issues/4405? i think marcie implemented something. what's the way forward?

anthrotype: i'd say we KISS and only set at build time if any of the glyph masters has overlaps

madigens: as in, we test for intersection of all against all segments?

anthrotype: we can also do a post-fix script that works on existing VF binaries, extracts the glyph would-be "masters" from the gvar tuples' peaks, instantiate at those location, and check for overlaps no, only check the masters if more complex than that, add a way for font dev to set it manually in sources via lib key no fancy product or combinations of axis value stops i think checking the masters covers most of the offenders in regular fonts

madigens: yeah i mean for font in fonts: for glyph in font: check_any_self_intersection(glyph)

anthrotype: for glyphs in zip(fonts): for glyph in glyphs: if any_glyph_overlap(glyph)

madigens: which comes down to decomposing glyphs into outlines-only (like skpath does) and checking the intersection of all segments against all others

anthrotype: you let skia do that

madigens: k

anthrotype: something like this can be used to extract the master locations for each glyph from gvar https://gist.github.com/anthrotype/a1498f5eefd747ffa47ae4d421e905a7 in case we want to apply this to an already built VF but I think we also want to be able to set the flags at build time and we should define a public.hasOverlaps lib key in the Glyph.lib maybe, not a big list in Font.lib whichever is more convenient I think the TTGlyphPen should be the one setting the bit while building the TTGlyph with an option maybe

madigens: yes. it should also keep track of bounds so we don't have to do it again later

anthrotype: actually, maybe a ufo2ft preProcessor can set the lib key on the UFO glyphs that contain overlap then the outlineCompiler will call TTGlyphPen with the option to set the flag when the lib key is present then the TTF masters will contain the glyf flags, and varLib.build can simply set it on the VF when any of the master TTFs has it the preProcessor should skip processing any glyph that already has the flag actually, maybe a global list in the font.lib is better in that sense, we can simply check if that's present we don't check anything and assume that list is complete if missing we do the costly overlap-checking public.glyphOverlaps or public.glyphsWithOverlaps maybe we can start with private ufo2ft lib key and publicize it later once we have impl

[m4rc1e]

@madig thanks for the paste and SGTM. I'm going to submit another test font before I start working on the implementation. I think Open Sans is too risky for the first family.

[chrissimpkins]

What's the status here @m4rc1e?

[chrissimpkins]

This issue impacts the Roboto family update as of the latest PR with variable font format files (https://github.com/google/fonts/pull/7231).

Moving it into Q2 and we'll address how to approach it in advance of landing the Roboto format update.

cc @anthrotype @rsheeter @davelab6

[chrissimpkins]

@emmamarichal Can I ask you to collaborate with Marc on this? Let's come up with a solution for Roboto that works across the rest of the catalog. It can be design outline changes, a compiler solution, a font editor solution, ...

[chrissimpkins]

@m4rc1e mind summarizing the current consensus on the approach to our overlap bit flag settings in this thread?