Add support for icon fonts

[johnspackman]

We need to integrate icon font support into qooxdoo-compiler, to match the generator as documented here : http://www.qooxdoo.org/5.1/pages/development/icon_fonts.html

[level420]

There exists a npm module for fontforge, but as far as I can see from https://github.com/sgen/node-fontforge the module is a simple wrapper for the command line. This would mean that fontforge has to be installed as a prerequisite. I'ts also unclear if the module is compatible with windows.

[cajus]

The question is whether the generator needs to be able to do it. Or if we only provide a "tool" that generates a qx contrib/package to be included. Not sure if it's a good idea to put these fontforge dependencies into it. Same for sprite maps, etc.

[level420]

@cajus what about https://github.com/devongovett/fontkit ? Does it have all you need to extract the glyph information?

[level420]

See https://github.com/devongovett/fontkit#glyph-metrics-and-layout

[cajus]

@level420 needs inspection. If id is the glyph name, then chances are good that it is possible.

But what I meant with my previous comment was: it doesn't seem to be easy enough to include a icon font set right now. It would be good to have a mechanism to: qx make-font-package FontAweSome.ttf and it will generate a package for inclusion. So that developers don't have to care. Just qx add _package_name_ and you're done.

@johnspackman if you tell me where to hook in, I can take care about the font extraction.

[cajus]

I'd also like to extend it to be able to create own font icon sets from a couple of SVG images. We're doing it already here, but it's too specific to be PRed in the moment. Maybe the JS way would be a good argument to make it a bit more usable for the rest of the world...

[level420]

I'm currently not familiar with qx-cli, but what would be best from my point of view is to just have a key in the config.json-equivalent file where I can add the icon font file, which then automatically does all is needed: load the font file with a decent name like FontAwesome with the qx loader and allow to reference a glyph in the icon image definition like qx.ui.form.Button('@FontAwesome/heart');. No extra font map files, no need to add the icon font into the fonts part of the appearance.

[level420]

As an addition it would be even greater to allow URLs to a cdn in the config file, where the font files are delivered from (like https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/fonts/fontawesome-webfont.*). They could then be directly used per URL or downloaded at generation time.

[cajus]

That wouldn't be a big deal.

[johnspackman]

@cajus it would fit into the ResourceManager class I think, and then it would work as @level420 suggested and be able to automatically compile - there's no stub for plug in conversions at present, I'll have to mock something up (maybe think about a plug in architecture) and get back to you ...

[cajus]

@johnspackman: I don't see what to put where in the compile.json file to trigger a font resource handler. If you want to use a webfont, it needs to be placed there in various formats (i.e. ttf, eof, woff, svg) to have a broad browser support. Just one of these files needs to be inspected to extract the glyph information/aspect ratio. Also svg would conflict with the ordinary image handler.

Maybe a convention like a "fonts" directory containing "fontname" subdirectories with all the files inside? Or configurable thru compile.json where one need to pass a mapping from fontname to the resource directory it's placed in?

[johnspackman]

Ah, it can't be inferred from the files on disk ... I was assuming that the resource manager would find "myfont.ttf" and so your plugin knows to convert it. But I guess that would mean that every font would get processed as a web font and that's not cool at all...

This needs to be in Manifest.json, not compile.json, so that when a library (eg a contrib) is included, the compiler automatically detects that the webfont needs to be compiled.

If you could write a class with an method that processes a font, ie something that's unit testable and meanwhile I'll arrange a hook into qxcompiler.Library. The method needs to be asynchronous, either return a Promise or accept a callback whichever you prefer

[cajus]

Ok. Not sure how to make it asynchronous. I guess the only method that runs asynchronously there is a non blocking file open/read. So if you're not running multi process, I'll be blocking anyway ;-)

Not sure what that function should pass back. Does it to do it, or do I need to feed the generated stuff somewhere else?

[cajus]

The input could be something like that if the data is pulled remotely:

  name : "FontAwesome",
  defaultSize : 40,
  resources : [
    'https://maxcdn.bootstrapcdn.com/font-awesome/4.7.0/fonts/fontawesome-webfont.eot?v=4.7.0',
    'https://maxcdn.bootstrapcdn.com/font-awesome/4.7.0/fonts/fontawesome-webfont.woff2?v=4.7.0',
    'https://maxcdn.bootstrapcdn.com/font-awesome/4.7.0/fonts/fontawesome-webfont.woff?v=4.7.0',
    'https://maxcdn.bootstrapcdn.com/font-awesome/4.7.0/fonts/fontawesome-webfont.ttf?v=4.7.0',
    'https://maxcdn.bootstrapcdn.com/font-awesome/4.7.0/fonts/fontawesome-webfont.svg?v=4.7.0#fontawesomeregular'
  ]
}

Is there a convention yet, where the downloads should be "cached"? Should be outside the source/resource folder I guess?!

[johnspackman]

everything goes into the target's output directory, its probably easier if I pass that to you. We want it to be configurable as an API, so using that example it would be perfect if you wrote a class that has properties for name, defaultSize, resources, etc plus outputPath and then leave it up to qooxdoo-cli to figure out how to configure it to make it work

[johnspackman]

also re blocking: there can be lots of things going on while you read the font - for example, downloading URLs and inspecting contents to determine which version to use, or whether to take a cached font file instead of downloading a new one can all happen while classes are being scanned or resources processed.

I'm throttling resource scanning at the moment otherwise the queues get too long (i.e. there's no point processing 1,000's of images at the same time), but making everything async has produced some really great results in throughput - when I first started testing, the generator was spending a huge proportion of it's time waiting for I/O to complete, but my early compiler was maxing out the CPU and never waiting for anything (that was when I could compile a skeleton from scratch including resources in something ridiculous like 1 second, compared to 3 minutes or more for the generator). Of course that was before adding babel for transpiling etc, but the difference of fully async was very striking

[cajus]

An additional thought: to make it really easy to use icon fonts, it should be enough to just add the definition to the manifest. To make this work, I'd need to be able to inject some bootstrap code that creates the required font entries automagically for the developer.

[oetiker]

since popular fonts like fontawesome do not contain 'good' glyph names in the font file, it would be great if there was a way to have such fonts with good fontmaps generated by other means provided as contribs ...

does the design allow for that ?

[johnspackman]

@cajus 👍 I completely agree, especially as adding contribs is now a simple command line, then the library Manifest needs to be able to boot strap any components or resources like webfonts. I'll add whatever you need to the boot loader, but I imagine that it will be fairly trivial, i.e. call a registry function for each webfont and then a notification for application startup (ultimately the notification could be done via a generic mechanism described in https://github.com/qooxdoo/qooxdoo/issues/9380, but in the mean time I'll patch something specific in)

@oetiker sounds reasonable, the design we have certainly doesn't prevent it

[cajus]

@oetiker: we've found that i.e. the SVG has 100% coverage for the glyph names in fontawesome. My current approach is to parse all parseable font files until the coverage is good enough. I agree that we should additionally add a key to provide a custom mapping from a symbolic name to unicode - if the font completely fails.

[oetiker]

@cajus cool ... (parsing several files) ... what I find especially confusing with fontawesome is that in the opentype version of the fontfile there are some names that are present but do not match the 'web names' ...

will you build a list of alternate names by parsing multiple fontfiles ?

[cajus]

@oetiker I'm currently experimenting with the fontkit to see what's possible with it. I can read the names and I've access to the post table. Do you have a name or alias handy which does not work in the current implementaiton?

[oetiker]

I only remember when I used it that some icons I found in http://fontawesome.io/icons/ did not exist ... I just checked my app and found that I just limited myself to the glyphs with proper names :)

[cajus]

Hehe :-) Looks that in case of fontawesome it does not help to scan all the files. It's not intended to be used that way. They want you to use the CSS. If you take a look at the ligature icon font, you see that it's solved much more elegantly. They only need the CSS for one single rule - which delivers the glyphs by glyph name.

In other words: fontawesome will need a special mapping file to make it work in the expected way. Maybe we'll create a package for that later on that generates it from here.

[cajus]

@johnspackman I've a class now that does asynchronusly work on a font definition and can be called like this:

const plug = new Fontify();
plug.name = "FontAwesome";
plug.defaultSize = 40;
plug.mapping = "fontawesome.map";
plug.resources = [
    'http://localhost:8000/fontawesome-webfont.ttf?v=4.7.0',
    'fontawesome-webfont.svg'
//    'https://maxcdn.bootstrapcdn.com/font-awesome/4.7.0/fonts/fontawesome-webfont.eot?v=4.7.0',
//    'https://maxcdn.bootstrapcdn.com/font-awesome/4.7.0/fonts/fontawesome-webfont.woff2?v=4.7.0',
//    'https://maxcdn.bootstrapcdn.com/font-awesome/4.7.0/fonts/fontawesome-webfont.woff?v=4.7.0',
//    'https://maxcdn.bootstrapcdn.com/font-awesome/4.7.0/fonts/fontawesome-webfont.ttf?v=4.7.0',
//    'https://maxcdn.bootstrapcdn.com/font-awesome/4.7.0/fonts/fontawesome-webfont.svg?v=4.7.0#fontawesomeregular'
];

plug.process().then(() => {
  console.log("### PROCESS done");
}).catch((err) => {
  console.log("### ERROR");
  console.log(err);
});

It currently just outputs the generated/predefined mapping to the console. And it does not provide any bootstrap code to be able to directly use a font within a freslhly bootstrapped qx application yet.

Just tell me where to plug in when the interface is basically ready. Note, that I'll be mostly offline until thursday.

[johnspackman]

brill, i've just pushed an update to qooxdoo-compiler (github only, not npm) that has a stub in there for webfonts - take a look at qxcompiler.WebFont. The library Manifest.json should look like this:

"provides" : 
  {
    "namespace": "mycontrib",
    "class": "source/class",
    "resource": "source/resource",
    "translation": "source/translation",
    "webfonts": [
        {
            "name": "FontAwesome",
            "defaultSize": 40,
            "mapping": "fontawesome.map",
            "resources": [
                "http://localhost:8000/fontawesome-webfont.ttf?v=4.7.0",
                "fontawesome-webfont.svg" 
              ]
        }
      ]
  }

(Unfortunately Manifest.json doesn't support comments yet)

The stub will just print out some text to say it's there; there are two methods, generateForTarget which will be called once overall and generateForApplication for every application - I'm not sure if the second is necessary but thought it might be useful. The only catch is that the generateForTarget is temporarily being called for every application

[cajus]

@johnspackman I don't see any output from the WebFont class. It doesn't seem to be called at all. I've added the "webfonts" key from your example above and updated cli/compiler. Anything else that needs to be adapted?

[cajus]

Maybe I just understand how to use cli/compiler from git. Did checkout and npm install and npm link (as root) in order to make it accessible.

[cajus]

Looks like it doesn't work because qx-cli doesn't use the checked out version of qooxdoo-compiler.

[johnspackman]

no - what you have to do is checkout both, and:

cd qooxdoo-compiler
npm install
npm link
cd ../qxoodoo-cli
npm install -g
npm link qxcompiler

This will make your git qooxdoo-cli global, and link it's qooxdoo-compiler module to your git one

[cajus]

Hmm. That's what I did initially (from the readme). Looks like I've to run the npm link qxcompiler after each npm install again.

[johnspackman]

cool, glad you got it working 👍

[cajus]

I've it working locally now. One little poll before finishing up:

There was the idea to make font based icons easier to use, i.e. just putting the definition in the Manifest would allow us starting to use @FontName/glpyh/size as Icon sources without any additional work. Having it in the Manifest means, that the font is available in all defined Themes - even if it's not used there. And it means that I've to patch the font manager to automatically include it when applying themes. I also need to temporarily store the font definition (i.e. in qx.$$iconFonts) until the theming is set up.

Is that OK and the way that we want to go here?

[oetiker]

If this means that it will be possible to ship Icon fonts with contrib themes, then yes! by all means!

[johnspackman]

ISWYM, that's a lot of work to go to if its not wanted. If the font was specified as an @asset so that its only included if necessary, but that won't work for CDN URLs (AFAIK the generator does not allow@asset to download resources automatically), and would mean that we would be assuming that all files with a given extension (woff, ttf, etc) are a webfont.

Im not sure it's a good idea to make that assumption because "ordinary" fonts are legitimate and as we support including third party libraries we'd be making assumptions about what they can contain.

As it stands, the compiler will only include resources from a contrib if at least one of the contrib classes are required, so by specifying the webfont in a contrib's Manifest it is possible to get some optimisation - just at the granularity of an entire library. This still means that for a large library like Qooxdoo itself, all applications would always have all webfonts that any of our themes use.

To solve this we could either move each theme into a separate contrib (not sure how much I like that idea, especially as themes share resources), or add a new @webfont() compiler hint. The downside of supporting @webfont would be that the generator would need to recognise it as an asset (presumably a minor change)

[cajus]

@johnspackman until we don't ship a default theme with icon fonts, it doesn't hurt. I like your idea of removing all these themes from the core package - except one simple one without external dependencies.

[cajus]

@oetiker that would be the idea, yes. So just core + your favorite theme package. The only drawback is if you've multiple themes in one application: if you switch the theme, both themes will have the webfont available.

[cajus]

097afddcdf4d98299e0ac2d348295fbf7accbd08 makes it work like in 5.x. e8762adf33ffea380d7a49d2a831f9472e7760ad adds custom font maps. 70371da8f2c639d94ddcc8bc8c4c5416e75d2ac2 enables automatic font inclusion when used with https://github.com/qooxdoo/qooxdoo/pull/9398.