Improve support for FreeType glyph rendering

[vroad]

This is the demo that renders fonts with FreeType on openfl-next. Text layout and rectangular packing is currently done on starling library though.

http://community.openfl.org/t/freetype-based-font-rendrer-for-opefl-next-and-starling-openfl/729

As I shown in this post, I have been trying to fix inefficiencies in my fork.

• Render glyphs on the fly, not "everything at once"
• Access font information without accessing glyph metrics
• Store glyphs in single texture as much as possible, using MaxRects bin pack
• Each texture may store cache from multiple fonts(or different font sizes)
• Even when we have multiple textures for font caching, the renderer still renders glyphs from the same texture with single draw call

First of all, we should not think that all glyphs can be rendered at once, because Japanese and Chinese have many characters. Doing this may cause the app to freeze.

[jgranick]

You may have noticed (perhaps not) that I've been working on fonts recently. This is the big missing piece for OpenFL "next", and I'd love to see it not only implemented in a basic form, but to really see a solution that gives us strong control over fonts moving forward.

It's a bit messy, but here's a sample that uses it:

https://github.com/openfl/lime-samples/blob/master/TextRendering/Source/Main.hx

I broke the Arabic text, perhaps because I'm doing it all based on character codes, and I think it has a lot of non-printing characters, though it does address a number of points

• It lets you render glyphs on the fly
• Font glyphs only render at one size, though, per texture
• Getting glyph metrics is separate from rendering glyph bitmaps
• I've exposed more font metrics

[vroad]

Thanks for your hard work, but it still seems to have problems...

• It fetches all kerning information at once, but this takes lots of time if you use Japanese/Chinese fonts. It should lets you fetch kerning information on the fly as well.
• It lets you render certain glyphs, but it will repeat the same rendering if the image is not big enough to contain all glyphs. In my implementation, Font class just returns the rendered image as "an array of image(s)".

[vroad]

And my RenderToImage implementation returns glyph metrics as well, since I was not sure if it returns the same glyph advance when hinting is applied.

[vroad]

Official FreeType doc states about "4. The effects of grid-fitting". http://www.freetype.org/freetype2/docs/glyphs/glyphs-3.html

[vroad]

Ah, for now We don't have a method for accessing kerning information without decomposing the font.

[jgranick]

I have found it difficult to work through this process -- it's not a simple task to come up with a good, simple (but flexible) API for this ;)

I felt that the font.renderGlyphs API helped because it gives you the power to render one letter only, or several. I think that (perhaps?) this could work better if you could pass an existing map of glyphs, like font.renderGlyphs (20, new GlyphSet ("more letters"), glyphs); where it could (theoretically) append the new glyphs to the existing buffer.

I had a similar idea with metrics vs. images, glyph.image and glyph.metrics are both null by default, but when you use either the method to render, or to get glyph metrics, it populates that field. If you could pass existing glyphs in, then it could help populate both (if someone desired) on the same object.

With Harfbuzz for complex text layout, I'm not sure if the metrics will be as important (since it access it itself) but on the flip side, maybe you need metrics but not the image.

I would like to modify the font.decompose method to accept a glyph range as well, maybe you only want to convert outlines for a few glyphs

[vroad]

If you use FreeType without harfbuzz, I think that it still makes sense to return glyph metrics with rendered images. It is redundant to call FT_Load_Glyph twice, since it loads glyph in the rendering process.

if you use harfbuzz for text layout, you only need to use FreeType as a font rasterizer, as you said.

Font.renderGlyphs automatically create glyph map for you, but is it really necessary? Shouldn't we let users do it by themselves?

[jgranick]

I was running into inefficiencies in the way this way that the Font render API was working, so I've just given it an overhaul.

Now, it is much simpler, everything works on the glyph index (instead of accepting "GlyphSet" objects). In order to get the list if glyph indices for a given string, you call Font.getGlyphs ("hello"); and it returns it back. Then we can do subsequent calls using the glyph index, much simpler :smile:

There is Font.renderGlyph to render one image only, and Font.renderGlyphs to render a sheet that contains all the glyphs you requested. Currently, it aligns it in a grid pattern, but I think you may be using a smarter algorithm for packing it, so perhaps we could implement something like that internally here for better sizing.

This is not exposed, but before, I was noticing that it would crash if I tried to process too many glyphs at once. It used to create anonymous objects per glyph, now I am using a binary format to populate a ByteArray no matter how many glyphs are rendered.

[vroad]

Good job! I think that we still have couple of things to consider though.

Including rendered image data in single ByteArray may not be good for performance, especially when you are rendering glyphs in big size. It's relying on ByteArray's resize ability, since we cannot predict how big the rendered image will be. It may need to copy entire memory block to another place, to make allocated memory contiguous.

If we separate image data from glyph data like this, we could get better performance.

• an array of rendered images (ByteArray)
• an ByteArray that only contains glyph data (can be allocated before rendering glyphs?)

On Node.js, you cannot construct UInt8Array with new UInt8Array(someByteArray), as we are using Node's Uint8Array directry on Node.js target. We could replace that with some linline function call, such as createUInt8Array(ba) or getUInt8Array(ba).

In my implementation, I was directly uploading to OpenGL texture from ByteArray without creating ImageBuffer or Image, but avoid creating these instances may not make huge performance difference.

[vroad]

No, appending header to each ByteArray is enough. So we only need to pass

• an array of rendered images + header(ByteArray)

As that data doesn't need to be resizable, we could use alloc_buffer instead of ByteArray for performance, but that may be not necessary either.

[vroad]

I meant "prepend", not append...

• an array of "header + rendered images"

I wonder if this change still cause some native targets to crush.

In your samples, you are not setting GL_UNPACK_ALIGNMENT. Since this value is 4 by default, without setting this value(or making the width of alpha-only image mutiple of 4) uploading alpha-only texture will fail and result in corrupted texture content.

https://github.com/openfl/lime-samples/blob/66739ef724132b85c97e978478141a66f1538874/TextRendering/Source/Main.hx#L153

[vroad]

Well, I didn't know that Font.renderGlyphs() still uses power of two image... Then I would use Font.renderGlyph() instead, as renderGlyphs would not allow me to use custom glyph map.

[vroad]

Looks like ByteArray doesn't support offsetting without creating copy. UInt8Array does copy in its constructor, too.

• resizing ByteArray several times
• Copy part of ByteArray to create image data
• Copy in UInt8Array Constructor (on Node.js we can avoid copying by obtaining byteView property)

Too many copies! Then separating image data from glyph data(width, height, x, y) would make more sense, as it reduces at least one copy.

[jgranick]

If it was sensible, we could consider separating the glyph info from the pixel data, have one big binary blob of only the glyph data, I'm not sure if that would be sensible to push into GL or if we should assume the one copy from the internal glyph render to generating the ImageBuffer instance

I'm not sure that ByteArray is as optimized as it should be, I definitely think the typed arrays should be improved. I had an implementation going that used a "DataView" class, and all the others (UInt8Array, Int32Array, etc) were abstracts over it. This allowed changes between one type of view to another without a cost. These are all interfaces over a single piece of data, so it should not be so expensive. It was about 2-3 times faster if I recall

[vroad]

I tried that by myself. On hxcpp and nodejs, TextRendering example didn't crush. On neko, app crashes in somewhere in the code. It doesn't matter how many number of letters I try to render, strange... Isn't this neko's bug? https://github.com/vroad/lime/commit/c41917519445198dc9a6c09a2bb9144c516a23c4

[vroad]

I reverted my changes, rebuild tools and ndll, but neko still crashes. Why??

[jgranick]

I noticed that Neko works if I use only English text, I'm guessing this is something with Neko and unicode character encoding

[larsiusprime]

Neko has consistently in the past been unable to render characters that cpp & flash targets had no problem rendering, so I suspect that's a solid line of inquiry.

On Sat, Mar 14, 2015 at 1:28 PM, Joshua Granick notifications@github.com wrote:

I noticed that Neko works if I use only English text, I'm guessing this is something with Neko and unicode character encoding

— Reply to this email directly or view it on GitHub https://github.com/openfl/lime/issues/368#issuecomment-80656942.

www.fortressofdoors.com -- Games, Art, Design

[vroad]

Even with latest development version of neko, TextRendering still crashes. This shoud be reported as a bug.

[vroad]

You fixed something in the latest version? Neko no longer crashes with current lime.

[vroad]

I still have doubt about API design of Font.renderGlyphs... It's not friendly to texture caching.

• It doesn't allow you to add glyphs to existing textures with glTexSubImage2D, since it returns results as single Image.
• It always search for duplicate glyph indices. Users may have their own glyph-index to texture-rectangle Map(and only want to render glyphs not in cache), so it's not always necessary.
• You need to lookup the returned map to obtain image sub-rectangle.

[vroad]

I think that we should have separate classes for bin packing. If you want single image that can be uploaded with glTexImage2D, you should create it manually by converting result images.

[jgranick]

Hey! Yeah! It does work in Neko. I'm not sure what changed to do that, but I'm happy for it.

I'm open to improving the Font API, though I'm aware that Lime shouldn't be too high level -- you're right -- I think there are smarter ways to pack textures (although I'm not sure if we should do multiple font sizes in the same atlas, unless we overwrite them? Just concerned about the ability to dump unused symbols later). The packing should probably occur in a higher level than Lime.

I'm not sure performance-wise if it would be better to call renderGlyph repeatedly or renderGlyphs and copy into a new texture

[vroad]

I modified lime and lime-samples with techniques I described above. It only supports one texture cache for now(since it was difficult to implement QuadBatch by myself), but still tries to render glyphs from one texture as much as possible.

https://github.com/vroad/lime-samples/commit/7c8c263d8d3f56ca5c1b606fa18b14b114dafd21 https://github.com/vroad/lime/commit/5b4965bd72850b62b64aa25f780cd7c32ad0924f

[vroad]

Do we really need to have a function that renders only one glyph? We should be able to do the same thing with Font.renderGlyphs, and it should be rare to render only one glyph alone. Whether it is significant or not, Font.renderGlyphs should have less overhead than multiple Font.renderGlyph.

There are still number of things to consider...

• Texture cache stores glyphs into the same texture regardless of font size. Is this really good for memory efficiency?
  • If default texture size is not big enough to contain next glyph, current implementation just creates the texture which will be able to contain at least that glyph.
  • MaxRectBinPack doesn't seem to support deleting existing rectangle. It is not possible to reuse part of texture cache. It is still possible to delete entire texture, though.

In the case of texture deletion, we should render TextField to another texture with FBOs, maybe.

Sadly neko crashes again after applying my changes, even though cpp and nodejs works without problems.

[vroad]

lime_gl_tex_sub_image_2d had the problem with null value. Now new TextRendering example should work on all native targets. https://github.com/openfl/lime/pull/400

[vroad]

Was the use of anonymous objects really cause of neko crash? I doubt it, because you didn't show where neko crashed and what fixed the problem.

I created a new pull request for it and few other improvements. https://github.com/openfl/lime/pull/419

[vroad]

We could keep using maps for storing rendered glyphs, though bin packing should get separated from renderGlyphs, shouldn't it?