Font fallback package fails under RTL

[deepakjois]

I am having this weird problem formatting numbers and latin text with using Noto NastaIiq Urdu. I am not sure if this strictly a problem with SILE, because the problem does not occur in Scheherazade font (see attached code and screenshot). However, this seems to work automatically with other software (text editors and word processors). So I am wondering what’s wrong, and whether it can be fixed in SILE .

Code:

\begin[class=urdu,papersize=a5]{document}
\set[parameter=document.parindent,value=0pt]
\nofolios
\font[family=Noto Nastaliq Urdu,style=Regular,language=urd,size=16pt]
اردو ہندی 1947 میں deepak آیا تھا

\font[family=Scheherazade,style=Regular,language=urd,size=16pt]
اردو ہندی 1947 میں deepak آیا تھا
\end{document}

Output

It would be a real bummer to format numbers and latin text by enclosing them in separate directives, and I seem to recall that this used to work.

Is this a problem with font packaging, or should it be considered a bug in SILE?

(P.S : Sorry for a slew of bug reports. I picked SILE up again after a hiatus to see if I could do some serious bilingual typesetting, and I keep running into issues)

[deepakjois]

@khaledhosny Could you help me understand what could be the problem here? It works with Amiri as well, but not with Noto Nastaliq Urdu. I recall you worked with both fonts.

[alerque]

This is a Noto font issue. The font you are using doesn't have glyphs for western alphabet characters. Scheherazade does—there is a baked in set of western glyphs at a size that matches the Arabic script. The reason it works for you in other systems is they are using a fallback font stack to give you glyphs from some default system font if they don't find one in the requested font, while SILE is trying to actually use the font you specified manually.

[alerque]

@deepakjois Amiri also has a Latin alphabet baked in (mostly copied from Crimson, a little bit from Libertine, and a little bit tweaked).

[deepakjois]

@alerque Thanks. I suppose a nice way to handle this would be to have a fallback mechanism in SILE as well.

Another nice way would have been if Noto Nastaliq bundled a bunch of Latin glyphs. Does anybody know if there is an easy way I could splice in Latin glyphs into Noto Nastaliq?

I will leave this here, but feel free to close if this is not considered a bug.

[khaledhosny]

What @alerque said. It is possible to merge two fonts together with pyftmerge from FontTools. It would be nice if SILE supported some sort of font fallback mechanism, it needs not be automatic (one could argue that it better not be automatic), but at least give the user way to select fallback fonts (a simple list, per script or per Unicode range, it will need some thought).

P.S. I developed Amiri, but I have no direct relation with the Noto project.

[deepakjois]

@khaledhosny Yes, I realise you are not directly involved with Noto. I recall seeing a comment from you googlei18n/noto-fonts#442 so I thought you might have a clue.

Anyway, thank you!

[simoncozens]

The answer is that SILE needs to have font fallback. The problem is determining when it's needed. I need Harfbuzz to tell me when it's getting notdef's back from the font. (I think it can do this, but I am unsure of how to make the interface work.)

[simoncozens]

I'm working on having font fallback work as an add-on package. I don't think it should be part of the core and on by default, because when typesetting you generally want the text you supply to be typeset in the fonts you supply, and if it can't be, that's a problem you need to know about. Explicitly matching fonts is better than implicitly doing so.

Here is the progress so far. What works:

\script[src=packages/font-fallback]
\font[family="Gentium,Symbola,Noto Sans CJK JP",style=Regular,size=16pt]
あば x 😼 Hello world. あ

What doesn't:

\font[family="Noto Nastaliq Urdu,Gentium",style=Regular,language=urd,size=16pt]
اردو ہندی 1947 میں deepak آیا تھا

In other words, my good friend bidirectionality.

[cmahte]

It would be nice if you could tell SILE to use various fonts for unicode ranges... I call that 'font configuration' or 'font substitution' and not 'fallback' where missing glyphs are presented in the first font found to contain the glyph.

As an example, if I have an italic Bengali paragraph, I'd like those western numbers to be in an italic font. Font fallback might not automatically find that, but a font substitution could point you to exactly the glyph you need.

On Wed, Nov 25, 2015 at 1:22 AM, Simon Cozens notifications@github.com wrote:

I'm working on having font fallback work as an add-on package. I don't think it should be part of the core and on by default, because when typesetting you generally want the text you supply to be typeset in the fonts you supply, and if it can't be, that's a problem you need to know about. Explicitly matching fonts is better than implicitly doing so.

Here is the progress so far. What works:

\script[src=packages/font-fallback]\font[family="Gentium,Symbola,Noto Sans CJK JP",style=Regular,size=16pt] あば x 😼 Hello world. あ

[image: screen shot 2015-11-25 at 16 21 01] https://cloud.githubusercontent.com/assets/106728/11390790/9478ccde-9390-11e5-995e-efba7b4fa2d8.png

What doesn't:

\font[family="Noto Nastaliq Urdu,Gentium",style=Regular,language=urd,size=16pt] اردو ہندی 1947 میں deepak آیا تھا

[image: screen shot 2015-11-25 at 16 21 44] https://cloud.githubusercontent.com/assets/106728/11390792/a50f034c-9390-11e5-8300-7e732c4956bf.png

In other words, my good friend bidirectionality.

— Reply to this email directly or view it on GitHub https://github.com/simoncozens/sile/issues/191#issuecomment-159523675.

[simoncozens]

Yes, I want that too. That's why bug #83 is still open. :-)

[simoncozens]

Wait, it strikes me that this isn't actually wrong. Given the input

\font[family="Noto Naskh Arabic",style=Regular,language=urd,size=16pt]

اردو ہندی 1947 میں deepak آیا تھا

\script[src=packages/font-fallback]
\font[family="Gentium,Noto Naskh Arabic",style=Regular,language=urd,size=16pt]

اردو ہندی 1947 میں deepak آیا تھا

I get the output:

@deepakjois or @khaledhosny, could you confirm this is what you expect to see?

If the word ordering is wrong (the two words between "1947" and "deepak" on input have been moved to the end of the line) then we have a bidi bug, orthogonal to the font fallback package. If the ordering is right, then I think font fallback is done.

[deepakjois]

Yes, the ordering is fine.

[simoncozens]

Hooray! Looks like we have font fallback (I should really call it font substitution) working. In fixing #279 I will also fix up support for specifying fonts by file name and also documenting the operation of the package, but I think we're done here.