Implement Universal Font Shaper

JimBobSquarePants commented 2 years ago

Prerequisites

[x] I have written a descriptive pull-request title
[x] I have verified that there are no overlapping pull-requests open
[x] I have verified that I am following matches the existing coding patterns and practice as demonstrated in the repository. These follow strict Stylecop rules :cop:.
[x] I have provided test coverage for my change (where applicable)

Description

A Work-In-Progress implementation of the Universal Shaping Engine (USE). Fixes #217

There's a fair amount to do here but I have some of the groundwork laid out already. I was hoping to base the work primarily on FontKit but I have no idea how to implement the state machine.

If anyone from the community is reading this PLEASE HELP OUT. I can't possibly figure this all out on my own ❤️.

Guide. https://github.com/n8willis/opentype-shaping-documents/blob/master/opentype-shaping-use.md

HarfBuzz Implementation https://github.com/harfbuzz/harfbuzz/blob/main/src/hb-ot-shaper-use.cc

FontKit Implementation https://github.com/foliojs/fontkit/blob/master/src/opentype/shapers/UniversalShaper.js

TODO:

[x] Refactor shapers to allow shaping stages.
[x] Add Indic shaping data.
[ ] Add USE specific Extended Indic shaping data.
[x] Implement state machine.
[x] Implement category lookups.
[x] Implement shaper.
[x] Add tests.

I decided to leave attempting to add USE specific Extended Indic shaping data for now as it breaks the working statemachine. I imagine migrating the generation code to contain all the latest harfbuzz changes will mitigate this but it's not worth the effort for now.

Here's some example rendering of Balinese content. Our output is 100% compliant.

codecov[bot] commented 2 years ago

Codecov Report

Merging #288 (ce4f8a3) into main (7816350) will increase coverage by 1%. The diff coverage is 93%.

@@           Coverage Diff           @@
##            main    #288     +/-   ##
=======================================
+ Coverage     83%     85%     +1%     
=======================================
  Files        225     234      +9     
  Lines      12807   13922   +1115     
  Branches    1837    1917     +80     
=======================================
+ Hits       10757   11838   +1081     
- Misses      1613    1631     +18     
- Partials     437     453     +16

Flag	Coverage Δ
unittests	`85% <93%> (+1%)`	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/SixLabors.Fonts/StreamFontMetrics.Cff.cs	`90% <0%> (ø)`
...e/Resources/IndicSyllabicCategoryTrie.Generated.cs	`100% <ø> (ø)`
...nicode/Resources/UniversalShapingTrie.Generated.cs	`100% <ø> (ø)`
src/SixLabors.Fonts/StreamFontMetrics.TrueType.cs	`97% <50%> (ø)`
...Tables/AdvancedTypographic/Shapers/ArabicShaper.cs	`89% <70%> (+<1%)`	:arrow_up:
...bors.Fonts/Tables/AdvancedTypographic/GPosTable.cs	`86% <71%> (+1%)`	:arrow_up:
...les/AdvancedTypographic/Shapers/UniversalShaper.cs	`81% <81%> (ø)`
src/UnicodeTrieGenerator/StateAutomation/INode.cs	`86% <86%> (ø)`
...Tables/AdvancedTypographic/Shapers/ShapingStage.cs	`90% <90%> (ø)`
src/SixLabors.Fonts/TextLayout.cs	`85% <91%> (-1%)`	:arrow_down:
... and 16 more

... and 2 files with indirect coverage changes

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

JimBobSquarePants commented 2 years ago

OK.

So I have the beginnings of a plan here....

I need to port the state machine used by JavaScript from here https://github.com/foliojs/dfa

Instead of using Pegjs as the PEG (Parsing Expression Grammar) parser generator I can use Pegasus.

However... The original sample really abuses type safety so I'm having real difficulty with my early porting experiments.

stefannikolei commented 1 year ago

However... The original sample really abuses type safety so I'm having real difficulty with my early porting experiments.

any should go to hell xD

i was looking for other implementations besides the js and c implementation of harfbuzz.

I found a rust port: https://github.com/RazrFalcon/rustybuzz/blob/master/src/complex/universal.rs

perhaps this helps

JimBobSquarePants commented 1 year ago

However... The original sample really abuses type safety so I'm having real difficulty with my early porting experiments.

any should go to hell xD

i was looking for other implementations besides the js and c implementation of harfbuzz.

I found a rust port: https://github.com/RazrFalcon/rustybuzz/blob/master/src/complex/universal.rs

perhaps this helps

That rust port has an open issue because it's missing a Ragel compiled state machine, the state machine is the bit I probably need most.

I've had a look at Ragel before, but I cannot make head no tail of it. Supposedly you can build a C# state machine using it.

https://github.com/adrian-thurston/ragel

JimBobSquarePants commented 1 year ago

First test passes in the state machine tests! Fingers crossed this is a sign of things to come.

JimBobSquarePants commented 1 year ago

General question, should we be commiting .generated. files or should we be generating them as part of the build?

Good question. It's definitely a manual process just now plus there's a bit of a chicken/egg situation going on with the UnicodeData class that references all the generated tries. I'm sure there's smarter ways to do it but I always end up regretting messing around with custom build config.

SixLabors / Fonts