Assorted `fontmake` problems when building Schticks (2-axis varfont)

twardoch commented 7 years ago

I’ve run into a number of problems with glyphsLib and fontmake when trying to build Schticks Text, a 2-axis variable font based on the STIX Two Text fonts (OFL). The fonts I’m talking about are in the https://github.com/twardoch/schticks-fonts-ofl, Schticks/Sources/25-gly-mm-width1 subfolder.

Configuration

Mac OS X 10.12.3
Glyphs 2.4.1 (965)
Homebrew-installed Python 2.7.13

I've just installed the devel fontmake by running in the repo:

pip install --user -r dev-requirements.txt .
pip install --user .

so I had:

booleanOperations-0.6.4
cu2qu-1.1.2.dev0
defcon-0.2.1:
fonttools-3.6.0
fontmake-1.1.1.dev0
glyphsLib-1.3.1.dev0
MutatorMath-2.0.2.dev0
ufo2ft-0.3.2.dev0

I also did it via:

pip install --user -r requirements.txt .
pip install --user .

So I had:

booleanOperations-0.6.4
cu2qu-1.1.1
defcon-0.2.1
fonttools-3.6.0
glyphsLib-1.3.0
MutatorMath-2.0.1
ufo2ft-0.3.1

In both configurations, when running fontmake like this:

fontmake -g SchticksText-MM.glyphs --verbose 'DEBUG' --keep-overlaps --production-names -o variable

on the SchticksText-MM.glyphs font in 25-gly-mm-width1/bak font from my https://github.com/twardoch/schticks-fonts-ofl repo, I get a number of problems.

Spurious illegal names

fontmake reports one illegal glyph name but such a glyph name is not in the font.

INFO:fontmake.font_project:Checking Glyphs source for illegal glyph names
WARNING:fontmake.font_project:Found 1 glyph names containing hyphens: B: -219
WARNING:fontmake.font_project:Replacing all hyphens with periods.

com.adobe.type.autohint data

The glyphs contain a custom com.adobe.type.autohint entry which makes glyphsLib fail. Stuff in userData that cannot be parsed should be ignored.

userData = {
com.adobe.type.autohint = <ICA8aGludFNldExpc3Q+CiAgICA8aGludHNldCBwb2ludFRhZz0iaHIwMCI+CiAgICAgIDxoc3RlbSBwb3M9Ii0xNjkiIHdpZHRoPSI1MyIgLz4KICAgICAgPGhzdGVtIHBvcz0iLTgiIHdpZHRoPSIxMTUiIC8+CiAgICAgIDxoc3RlbSBwb3M9IjQ2MSIgd2lkdGg9IjIwOCIgLz4KICAgICAgPGhzdGVtIHBvcz0iNjMxIiB3aWR0aD0iMzgiIC8+CiAgICAgIDxoc3RlbSBwb3M9IjczNCIgd2lkdGg9IjUxIiAvPgogICAgICA8dnN0ZW0gcG9zPSIxMCIgd2lkdGg9IjUxIiAvPgogICAgICA8dnN0ZW0gcG9zPSIxMjYiIHdpZHRoPSI3NSIgLz4KICAgICAgPHZzdGVtIHBvcz0iMTI2IiB3aWR0aD0iOTYiIC8+CiAgICAgIDx2c3RlbSBwb3M9IjI0MyIgd2lkdGg9IjExNSIgLz4KICAgICAgPHZzdGVtIHBvcz0iMjgwIiB3aWR0aD0iNDAiIC8+CiAgICAgIDx2c3RlbSBwb3M9IjM5MSIgd2lkdGg9IjkwIiAvPgogICAgICA8dnN0ZW0gcG9zPSI1NDMiIHdpZHRoPSI1MiIgLz4KICAgIDwvaGludHNldD4KICA8L2hpbnRTZXRMaXN0Pgo=>;
};

Traceback

INFO:glyphsLib:Parsing .glyphs file
Traceback (most recent call last):
  File "/Users/adam/Library/Python/2.7/bin/fontmake", line 11, in <module>
    load_entry_point('fontmake', 'console_scripts', 'fontmake')()
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/Lib/fontmake/__main__.py", line 171, in main
    project.run_from_glyphs(glyphs_path, **args)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/Lib/fontmake/font_project.py", line 368, in run_from_glyphs
    glyphs_path, family_name)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/fonttools/Lib/fontTools/misc/loggingTools.py", line 372, in wrapper
    return func(*args, **kwds)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/Lib/fontmake/font_project.py", line 93, in build_master_ufos
    family_name=family_name)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/glyphslib/Lib/glyphsLib/__init__.py", line 91, in build_masters
    filename, include_instances=True, family_name=family_name)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/glyphslib/Lib/glyphsLib/__init__.py", line 64, in load_to_ufos
    data = load(file_or_path)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/glyphslib/Lib/glyphsLib/__init__.py", line 44, in load
    return loads(fp.read())
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/glyphslib/Lib/glyphsLib/__init__.py", line 53, in loads
    data = p.parse(value)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/glyphslib/Lib/glyphsLib/parser.py", line 43, in parse
    result, i = self._parse(text, 0)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/glyphslib/Lib/glyphsLib/parser.py", line 55, in _parse
    return self._parse_dict(text, i)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/glyphslib/Lib/glyphsLib/parser.py", line 83, in _parse_dict
    res[name], i = self._parse(text, i)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/glyphslib/Lib/glyphsLib/parser.py", line 61, in _parse
    return self._parse_list(text, i)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/glyphslib/Lib/glyphsLib/parser.py", line 104, in _parse_list
    list_item, i = self._parse(text, i)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/glyphslib/Lib/glyphsLib/parser.py", line 55, in _parse
    return self._parse_dict(text, i)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/glyphslib/Lib/glyphsLib/parser.py", line 83, in _parse_dict
    res[name], i = self._parse(text, i)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/glyphslib/Lib/glyphsLib/parser.py", line 55, in _parse
    return self._parse_dict(text, i)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/glyphslib/Lib/glyphsLib/parser.py", line 83, in _parse_dict
    res[name], i = self._parse(text, i)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/glyphslib/Lib/glyphsLib/parser.py", line 70, in _parse
    self._fail('Unexpected content', text, i)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/glyphslib/Lib/glyphsLib/parser.py", line 145, in _fail
    raise ValueError('%s:\n%s' % (message, text[i:i + 79]))
ValueError: Unexpected content:
 <ICA8aGludFNldExpc3Q+CiAgICA8aGludHNldCBwb2ludFRhZz0iaHIwMCI+CiAgICAgIDxoc3Rlb

Non-integer underline position

The font contains a weird entry:

name = underlinePosition;
value = -77.5;

which makes glyphsLib fail. I don't know why this entry is there but glyphsLib should round, not fail.

Traceback

INFO:glyphsLib:Parsing .glyphs file
INFO:glyphsLib:Casting parsed values
Traceback (most recent call last):
  File "/Users/adam/Library/Python/2.7/bin/fontmake", line 11, in <module>
    load_entry_point('fontmake', 'console_scripts', 'fontmake')()
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/Lib/fontmake/__main__.py", line 171, in main
    project.run_from_glyphs(glyphs_path, **args)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/Lib/fontmake/font_project.py", line 368, in run_from_glyphs
    glyphs_path, family_name)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/fonttools/Lib/fontTools/misc/loggingTools.py", line 372, in wrapper
    return func(*args, **kwds)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/Lib/fontmake/font_project.py", line 93, in build_master_ufos
    family_name=family_name)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/glyphslib/Lib/glyphsLib/__init__.py", line 91, in build_masters
    filename, include_instances=True, family_name=family_name)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/glyphslib/Lib/glyphsLib/__init__.py", line 64, in load_to_ufos
    data = load(file_or_path)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/glyphslib/Lib/glyphsLib/__init__.py", line 44, in load
    return loads(fp.read())
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/glyphslib/Lib/glyphsLib/__init__.py", line 55, in loads
    cast_data(data)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/glyphslib/Lib/glyphsLib/casting.py", line 510, in cast_data
    _convert_data(data, True, _TYPE_STRUCTURE)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/glyphslib/Lib/glyphsLib/casting.py", line 526, in _convert_data
    _convert_data(cur_data, to_typed, cur_type)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/glyphslib/Lib/glyphsLib/casting.py", line 528, in _convert_data
    data[key] = cur_type.convert(data[key], to_typed)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/glyphslib/Lib/glyphsLib/casting.py", line 78, in convert
    return self.read(data) if to_typed else self.write(data)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/glyphslib/Lib/glyphsLib/casting.py", line 309, in read
    value = int(value)
ValueError: invalid literal for int() with base 10: '-77.5'

TypeError: int() argument must be a string or a number, not 'list'

After I manually cleaned the offending portions in the .glyphs files using a text editor, and stored the font in 25-gly-mm-width1, then run fontmake on it, I get:

TypeError: int() argument must be a string or a number, not 'list'

I have no idea what to do with this, so I cannot proceed any further.

Traceback

INFO:glyphsLib:Parsing .glyphs file
INFO:glyphsLib:Casting parsed values
Traceback (most recent call last):
  File "/Users/adam/Library/Python/2.7/bin/fontmake", line 11, in <module>
    load_entry_point('fontmake', 'console_scripts', 'fontmake')()
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/Lib/fontmake/__main__.py", line 171, in main
    project.run_from_glyphs(glyphs_path, **args)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/Lib/fontmake/font_project.py", line 368, in run_from_glyphs
    glyphs_path, family_name)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/fonttools/Lib/fontTools/misc/loggingTools.py", line 372, in wrapper
    return func(*args, **kwds)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/Lib/fontmake/font_project.py", line 93, in build_master_ufos
    family_name=family_name)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/glyphslib/Lib/glyphsLib/__init__.py", line 91, in build_masters
    filename, include_instances=True, family_name=family_name)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/glyphslib/Lib/glyphsLib/__init__.py", line 64, in load_to_ufos
    data = load(file_or_path)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/glyphslib/Lib/glyphsLib/__init__.py", line 44, in load
    return loads(fp.read())
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/glyphslib/Lib/glyphsLib/__init__.py", line 55, in loads
    cast_data(data)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/glyphslib/Lib/glyphsLib/casting.py", line 510, in cast_data
    _convert_data(data, True, _TYPE_STRUCTURE)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/glyphslib/Lib/glyphsLib/casting.py", line 526, in _convert_data
    _convert_data(cur_data, to_typed, cur_type)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/glyphslib/Lib/glyphsLib/casting.py", line 528, in _convert_data
    data[key] = cur_type.convert(data[key], to_typed)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/glyphslib/Lib/glyphsLib/casting.py", line 78, in convert
    return self.read(data) if to_typed else self.write(data)
  File "/Users/adam/Developer/vcs/github/googlei18n/fontmake/src/glyphslib/Lib/glyphsLib/casting.py", line 136, in read
    return int(src)
TypeError: int() argument must be a string or a number, not 'list'

anthrotype commented 7 years ago

@twardoch thanks for the detailed bug report.

About the parser error issue with the custom userData, the issue should go away if you wrap the string value with quotes com.adobe.type.autohint = "<ICA8aGludFNldExp..>".

The glyphsLib parser accepts any character if it is between double quotes (the quote character iteself has to be escaped with backslash); whereas if it is outside of quotes, it only accepts characters [-_./$A-Za-z0-9]+. Here is the regex used: https://github.com/googlei18n/glyphsLib/blob/700d96cd9c546c20208606123f34806bb6156a79/Lib/glyphsLib/parser.py#L29

I wonder how that piece of data was written into the glyphs source file? Did Glyphs write it there, or some external tool?

anthrotype commented 7 years ago

the last traceback, TypeError: int() argument must be a string or a number, not 'list', has to do with the fact that your glyphs file contains color definitions such as:

color = (255,153,153,1);

But glyphsLib is expecting color definitions to be integers:

https://github.com/googlei18n/glyphsLib/blob/4b40d7f0ee34e8637069afb1b6776e4a9d43c6b9/Lib/glyphsLib/casting.py#L473

The inline comment says that this attribute is undocumented, and indeed I couldn't find any references to it in https://github.com/schriftgestalt/GlyphsSDK/blob/master/GlyphsFileFormat.md

maybe @schriftgestalt could tell us how glyphs' colors should be encoded in the source file?

I just tried to set the color of a glyph, saved the file and opened it up in a text editor, and the color is encoded as a decimal integer.

Could it be that the tuple variant was produced by earlier versions of Glyphs?

anthrotype commented 7 years ago

@twardoch can you tell us how that unquoted string com.adobe.type.autohint = <ICA8aGludFNldExp..> was written in the glyphs source file? If it was written by Glyphs.app itself, then we will need to adapt the glyphsLib parser to somehow accept such notation.

@schriftgestalt may I ask again how should glyph colors be encoded? As integers (like glyphsLib expects) or as (RGBA?) tuples like in Adam's .glyphs file?

davelab6 commented 7 years ago

Was this glyphs file generated with fl6? 😂

On Feb 11, 2017 1:14 PM, "Cosimo Lupo" notifications@github.com wrote:

@twardoch https://github.com/twardoch can you tell us how that unquoted string com.adobe.type.autohint = was written in the glyphs source file? If it was written by Glyphs.app itself, then we will need to adapt the glyphsLib parser to somehow accept such notation.

@schriftgestalt https://github.com/schriftgestalt may I ask again how should glyph colors be encoded? As integers (like glyphsLib expects) or as (RGBA?) tuples like in Adam's .glyphs file?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/googlei18n/fontmake/issues/247#issuecomment-279164802, or mute the thread https://github.com/notifications/unsubscribe-auth/AAP9y0m-wq2MdrHtwj0EwhHyI-UH7_MUks5rbfp4gaJpZM4L3cMy .

schriftgestalt commented 7 years ago

Color can be encoded as integer or as RGBA tupel.

The color and autohint info most likely comes from a .ufo file that was opened in Glyphs. The RGBA storage is mostly to be able to round trip .ufo.

anthrotype commented 7 years ago

I see, thanks Georg.

But shouldn't Glyphs.app wrap those userData strings with quotes "?

glyphsLib parser is choking because it only recognizes as a valid "value" token a string comprised of one or more [-_./$A-Za-z0-9] characters, unless that string is included in quotes, in which case .* anything matches.

https://github.com/googlei18n/glyphsLib/blob/700d96cd9c546c20208606123f34806bb6156a79/Lib/glyphsLib/parser.py#L29

value_re = r'(".*?(?<!\\)"|[-_./$A-Za-z0-9]+)'

If this is not correct, then we should modify the parser.

schriftgestalt commented 7 years ago

the value in com.adobe.type.autohint is stored as data and that is enclosed in <>. So it is not a string that needs to be enclosed in quotes.

anthrotype commented 7 years ago

Ok thanks. So it looks like the glyphsLib parser does not properly support binary data values in ASCII plist:

https://developer.apple.com/library/content/documentation/Cocoa/Conceptual/PropertyLists/OldStylePlists/OldStylePLists.html

NSData Binary data is enclosed in angle brackets and encoded in hexadecimal ASCII. Spaces are ignored. > For example:
<0fbd777 1c2735ae>

I'll file a separate issue over there.

twardoch commented 7 years ago

@davelab6 asked:

Was this glyphs file generated with fl6? 😂

I wrote in my report:

Configuration:

Mac OS X 10.12.3

Glyphs 2.4.1 (965)

Homebrew-installed Python 2.7.13

twardoch commented 7 years ago

Ps. I shall add that the .glyphs file did indeed originate from opening UFO files which were generated with vfb2ufo from a VFB saved in FontLab Studio 5. I used FontLab VI at an earlier stage. The folder list should give you an idea about the sequence of steps involved.

googlefonts / fontmake