[makeotfexe] feature parameter offset too large

[BoldMonday]

(Some of you might already be familiar with this because I have been in touch personally before.)

We have a typeface family, Logical, where every font contains a large number of substitution rules. We are talking about more than 10,000 rules spread over different features. All these features use the useExtension syntax. When I try to compile these fonts with the latest release (3.0.1) I get the following error:

makeotf [Note] Using features file at './features'.
makeotfexe [FATAL] <Logical> feature parameter offset too large (11cda)
makeotf [Error] Failed to build output font file 'Logical.otf'.

A long time ago (before the afdko became a github project) we were in contact with @readroberts about this problem and he was kind to supply a custom build version of makeotfexe that somehow was able to compile these features. But this custom build had some other problems in turn that never got resolved unfortunately.

So I am looking for a way now to compile these fonts 100% correctly with an official release version of the afdko. But I am unsure whether this is a bug that can be resolved or whether we are pushing makeotfexe too far. In case of the latter I am looking for a way to determine how many rules are the limit.

Any advice is greatly appreciated. And I am happy to share the files in question of course.

[BoldMonday]

So... Anyone? Pinging the other Adobe folks @miguelsousa @frankrolf @cjchapman Thanks.

[cjchapman]

@BoldMonday can you send us the files in question? Thanks.

[BoldMonday]

@cjchapman Email sent. Thx!

[readroberts]

@BoldMonday Is there a prior version of makeotf, or another tool, which builds this font? I do recall the custom version of makeotfexe circa Nov 2018 that could build this font. However, it did so only because it did not detect some overflow problems, and the built font was not correct. See our email thread 'Re: [makeotfexe] Problems with named stylistic sets and "useExtension" lookups' from Jan 30 2109 to Feb 28 2019; the last one is where I reported my conclusions that the font was just too large to be built.

I did investigate again because I did not expect the overflow to be in a FeatureParam offset, and found more information.

The biggest problem, which I do not see a way to resolve, is that that the useExtension keyword does much less for contextual lookups than for GPOS kern lookups. This has two parts. First, each contextual rule requires its own subtable, and consequently needs an offset to the subtable in the lookup record. Second, an Extension subtable can be applied only to a single subtable in a LookupRecord. This means that if you have a contextual lookup with 6000 contextual rules, and you apply the feature file useExtension keyword to the lookup definition, you will have a lookup record with a list of 6000 offsets to extension subtables. This is better for font size than 6000 Contextual rule subtables, as an Extension subtable is only 8 bytes, but in your font, you get an overflow either way because the LookupList is so large.

What's needed is a new Contextual look type that can reference a list of Contextual rule subtables. However, this would a require a change to the OpenType spec , and that all OpenType layout systems would have to change. This is not a problem for most lookup types, such as GPOS kern lookups, because each lookup subtable already can contain many rules. It is a problem unique to the contextual lookup type that each subtable can contain only one rule.

makeotf does have two issues that can cause an unnecessary overflow but which won't help with your font. First, Feature Parameter data is written after the LookupList, which is why the offset overflows. I will change this to be written right after the feature record which references it, as fontmake does. Second,, the aalt lookup record is sorted to the top of the LookupList, but its subtables are written after all the other directly referenced subtables, making it a candidate for overflow. I will fix this as well.

[BoldMonday]

Thank you for the support @cjchapman and for the explanation and bugfixes @readroberts. I am looking forward to try the new version.

I understand that our fonts are currently pushing makeotfexe too far and that we will have to trim the amount of contextual substitutions.

I was hoping there would be a way to determine how far it can be pushed. Some way to measure how much the features need to be trimmed. Is there anything that I can deduct from the current error message makeotfexe [FATAL] <Logical> feature parameter offset too large (11cda)?

[readroberts]

No, unfortunately. Both makeotfexe and fontmake simply report the first overflow they encounter, and there is no way to determine if there was some upstream declaration that is the primary cause, or if there would be a lot more overflows if we ignored the first. What's needed is to make offset overflows during the fill phase non-fatal, and change all the offset fields to long rather shorts so they can hold the data, and then add logic to report the layout table structures. This is not that hard, but probably about three days of work. Right now, the best you can do is to eliminate rules by hand until the font builds. I was able to make it work by adding useExtension to the aalt feature and cutting the set of rules in the ss04 down to about 320 lines. With the new version, once the fixed branch has been merged and released, you can probably have some more rules, but far from the full set. Once the font has built, you can use 'spot -t GPSO/GSUB' to do a structure level dump, and write a python script to look at all the subtable offsets and figure which features and lookups take how much space.

Note that I fixed only the issue with FeatParam. I did not fix the issue with the aalt subtables being written at the end of the GPOS/GSUB subtable list because a) it would take a major rework of the makeotfexe logic, and b) a better solution already exists: applying the useExtension keyword in the aalt feature definition reduces aalt's contribution of offset overflow even more.

[readroberts]

Also, the overflow issue with contextual rules is not just with makeotf's logic. Because the limitations of applying extension subtables to contextual rules, there is simply no way to pack more than about 8000 contextual rules in a GSUB or GPOS layout table.