Unicode math improvements

[vlasakm]

• [x] Don't set math families 2 and 3 for Unicode math

Setting math families 2 and 3 doesn't do anything special for "new math fonts"

See mlist.c:fixup_math_parameters in LuaTeX source:

https://github.com/TeX-Live/luatex/blob/eee644d052c295920d378ef579a96fcca497af9a/source/texk/web2c/luatexdir/tex/mlist.c#L478

• [ ] Use math family 0 for Unicode math

This allows shorter notation (i.e. 0 is implicit) when using \Umathchardef, etc.

This would have also prevented the former bug in \_setunimathdimens which is executed before the \mathcode for numbers are set:

  \_delimitershortfall=0.5\_fontdimen6\_textfont3
  \_nulldelimiterspace=0.12\_fontdimen6\_textfont3
  \_scriptspace=0.05\_fontdimen6\_textfont3
-  {\_everymath{}\_global\_setbox0=\_hbox{$\_displaystyle{0\_atop0}$}}% correction for \choose
+  {\_everymath{}\_global\_setbox0=\_hbox{$\_fam1\_displaystyle{0\_atop0}$}}% correction for \choose
  \_Umathfractiondelsize\_displaystyle = \_dimexpr(\_ht0-\_Umathaxis\_displaystyle)*2\_relax

}

• [ ] Possibly more, like:
  • Using the Unicode math mechanism even for old style TeX fonts. This allows having a single code path and should allow https://tex.stackexchange.com/questions/576840/font-extensions-and-conversion.
  • More that comes to my mind?

@olsak What is your plan on next release? I thought that with the amount of breaking changes in this release, we might be able to squeeze in more, so that there aren't several subsequent releases that break things. Possibly labeling the release 2.00, so that it is apparent that there are breaking changes?

Something else I also wanted to try are \_pagedest and \pgref, and in the spirit of Unicode math only, fonts loaded with \initunifonts only (no preloaded fonts).

[olsak]

Ad 1 (no math families 2 and 3): OK Ad 2 (family 1->0). I don't plan to change default Unicode math family which is 1 in OpTeX. Please, don't waste time with this.

[olsak]

Next release will be 1.04. I plan to do it in a week.

[vlasakm]

Ad 2 (family 1->0). I don't plan to change default Unicode math family which is 1 in OpTeX. Please, don't waste time with this.

Ok, but it seems like a weird choice. Not only because \_normalmath starts with 0 and \_normalunimath with 1...

Considering that \Umathcode <char> = 0 0 <char> is the default, then having the base font family in 0 means that there is a kind of reasonable default for every character. Now if some character is not covered by the tables it will be typeset from the preloaded cmr font, which will probably not work out that well, but the problem may not be apparent. This was the issue in https://github.com/olsak/OpTeX/issues/21.

I also planned to address the fact, that OpTeX trick 0030, math-unicode.opm and optex-math.pdf are incosistent about which family numbers are reserved.

@olsak Is there other reason, except for backwards compatibility, for keeping the family number as 1? The next release will not be completely backwards compatible anyways, so this may be a rare chance to do more.

[vlasakm]

OK, I stil think that family number 0 would be better. But not making that change now.

While messing with the Unicode math fonts I found other things addressed in the commits above:

• As already mentioned, setting math families 2 and 3 doesn't do anything special for "new math fonts"
• \scriptspace the script space setting didn't do anything, as far as I could tell. In the LuaTeX manual it is mentioned as a special case for TFM fonts. In short, \scriptspace value is only read if a formula is to be typeset, and \Umathspaceafterscript is not set. Which doesn't happen, since in OpTeX a formula is typeset only after Unicode math font is loaded. So even now the setting of \Umathspaceafterscript came from the font, (different values for for the different math styles).
• XeTeX style options starting at 1 were used for ssty font feature. I find the LuaTeX way much clearer.
• As in https://github.com/olsak/OpTeX/pull/7, the bbold font used optical sizes that don't really exist in .pfb files.

Also more thing perhaps worth addressing:

• While the relative size of script style and script script style is determined by OpTeX macros, actually Unicode Math fonts contain the ratios. Some examples:

latinmodern-math:
["ScriptPercentScaleDown"]=70,
["ScriptScriptPercentScaleDown"]=50,

firamath-regular.lua
["ScriptPercentScaleDown"]=72,
["ScriptScriptPercentScaleDown"]=58,

The ConTeXt fontloader has the option to use those ratios to scale down the fonts automatically. The OpenType spec suggests values 80 and 60. It also mentions that when ssty font feature is used the font designer can assume these ratios are used.

The ssty font feature also mentions the values 1 for script style and 2 for script script style, which AFAIK make more sense in the OpenType substitution handling and feel more natural. Hence also the commit above.

[vlasakm]

I also planned to address the fact, that OpTeX trick 0030, math-unicode.opm and optex-math.pdf are incosistent about which family numbers are reserved.

I am leaving this part to you @olsak. I don't feel like making decision about this. Some parts of documentation say that families 0..15 are reserved by OpTeX, other say that the user is free to use 5 and above. \_famalloc starts allocating at 4, but is redefined by math-preload.opm to start at 10 (\_count18=9). The mentioned OpTeX trick doesn't change the maximum allocation number for \newfam to 100... Maybe more I missed.

This is what I had before I gave up.

diff --git a/optex/base/alloc.opm b/optex/base/alloc.opm
index 13252ca..aae5f67 100644
--- a/optex/base/alloc.opm
+++ b/optex/base/alloc.opm
@@ -28,7 +28,7 @@
 \_countdef\_toksalloc=15   \_toksalloc=255
 \_countdef\_readalloc=16   \_readalloc=-1
 \_countdef\_writealloc=17  \_writealloc=-1
-\_countdef\_famalloc=18    \_famalloc=3
+\_countdef\_famalloc=18    \_famalloc=15 % 0..15 reserved

    \_doc -----------------------------
    The common allocation macro
diff --git a/optex/base/math-preload.opm b/optex/base/math-preload.opm
index 0ec9aba..3a60193 100644
--- a/optex/base/math-preload.opm
+++ b/optex/base/math-preload.opm
@@ -39,7 +39,6 @@
 %  \_setmathfamily 11 \_tenbi
   \_setmathdimens
 }
-\_count18=9   % families declared by \newfam are 12, 13, ...

 \_def \normalmath {\_normalmath}  \_def\boldmath {\_boldmath}

diff --git a/web/optex-tricks.html b/web/optex-tricks.html
index b23479e..715ce5c 100644
--- a/web/optex-tricks.html
+++ b/web/optex-tricks.html
@@ -1327,7 +1327,7 @@ macros follows:

 \def\addUmathfont #1#2#3#4#5#6{% #1: fam (will be set), #2#3: normal font, #4#5: bold font
    \ifx\_ncharrmA\undefined \errmessage{basic Unicode math font must be loaded first}\fi
-   \global\advance \famaddress by1
+   \incr \famaddress
    \global\chardef #1=\famaddress
    \global\addto\_normalmath{\_corrmsize#6 \_loadumathfamily #1 {#2}{#3} }%
    \ifx\relax#4\relax
@@ -1339,6 +1339,7 @@ macros follows:
    \wterm{MATH-FONT: added #1=\the#1, normal: "#2", \ifx"#4"\else bold: "#4"\fi}
 }
 \newcount\famaddress  \famaddress=100
+\chardef\_maifam = 100
 </pre>

 <p>The \addUmathfont has six parameters:

[olsak]

While the relative size of script style and script script style is determined by OpTeX macros, actually Unicode Math fonts contain the ratios.

Do you mean that we can read the data from OpenType math font and use it in the \_mathfontsize macro? How?

[vlasakm]

Do you mean that we can read the data from OpenType math font and use it in the \_mathfontsize macro? How?

@olsak The ConTeXt font loader already reads the math table, but it doesn't apply the ScriptPercentScaleDown parameters unless explicitly requeseted with the mathsize font feature. Because the specified size is multiplied by the scale parameters, same size has to be specified for all styles.

\initunifonts
\nopagenumbers
\tracingassigns=2

\def\test{%
  \textfont        10=\text
  \scriptfont      10=\script
  \scriptscriptfont10=\scriptscript
%
  $\fam10 {a_{\vrule b}}_{\vrule c}$
}

\font\text        =file:firamath-regular.otf:mode=base;script=math;ssty=0 at 100pt
\font\script      =file:firamath-regular.otf:mode=base;script=math;ssty=1 at 70pt
\font\scriptscript=file:firamath-regular.otf:mode=base;script=math;ssty=2 at 50pt
\test
%
%
\font\text        =file:firamath-regular.otf:mode=base;script=math;ssty=0;mathsize=1 at 100pt
\font\script      =file:firamath-regular.otf:mode=base;script=math;ssty=1;mathsize=2 at 100pt
\font\scriptscript=file:firamath-regular.otf:mode=base;script=math;ssty=2;mathsize=3 at 100pt
\test

\bye

I used Fira Math for demonstration (it has ScriptPercentScaleDown=72 and ScriptScriptPercentScaleDown=58), so 72pt and 58pt for the respective script/scripscript sizes. The base lines stay the same, while the script/scriptscript letters are (slightly) bigger in the right example. Also Fira doesn't have different shapes for most glyphs with ssty, so that is basically useless in the example above (just included for completeness).

I actually don't know how well these parameters are actually used/tested in LaTeX / ConTeXt and other OpenType Math systems. I couldn't find use of the font feature in proper ConTeXt (but it is very likely that I missed something) and didn't even look into LaTeX, nor XeTeX or MathJax how they deal with this.

[vlasakm]

Maybe more apparent difference with XITSMath-Regular.otf:

[vlasakm]

I have the following for use within OpTeX:

--- a/optex/base/math-unicode.opm
+++ b/optex/base/math-unicode.opm
@@ -127,16 +127,15 @@
 \_def\_mfontfeatures{mode=base;script=math;}

 \_def\_loadumathfamily #1 #2#3 {%
-  \_edef\_optsizesave{\_the\_optsize}%
-  \_optsize=\_sizemtext  \_font\_mF=\_umathname{#2}{\_textmff #3} at\_optsize
-  \_textfont#1=\_mF
-  \_optsize=\_sizemscript \_font\_mF=\_umathname{#2}{\_scriptmff #3} at\_optsize
-  \_scriptfont#1=\_mF
-  \_optsize=\_sizemsscript \_font\_mF=\_umathname{#2}{\_sscriptmff #3} at\_optsize
-  \_scriptscriptfont#1=\_mF
-  \_optsize=\_optsizesave \_ptmunit=\_ptunit
+  \_font\_mF=\_umathname{#2}{\_textmff    #3} at\_sizemtext \_textfont        #1=\_mF
+  \_font\_mF=\_umathname{#2}{\_scriptmff  #3} at\_sizemtext \_scriptfont      #1=\_mF
+  \_font\_mF=\_umathname{#2}{\_sscriptmff #3} at\_sizemtext \_scriptscriptfont#1=\_mF
 }
-\_def\_textmff{ssty=0;} \_def\_scriptmff{ssty=1;} \_def\_sscriptmff{ssty=2;}
+% ssty:     0 = text, 1 = script, 2 = scriptscript
+% mathsize: 1 = text, 2 = script, 3 = scriptscript
+\_def\_textmff   {ssty=0;mathsize=1;}
+\_def\_scriptmff {ssty=1;mathsize=2;}
+\_def\_sscriptmff{ssty=2;mathsize=3;}

    \_doc -----------------------------
    Unicode math font includes all typical math alphabets together, user needs not to

Not sure what the optsize business is about.

[vlasakm]

While thesting the above, I also realized that the set Unicode math parameters are taken from the last set math family for said style.

E.g. with\fontfam[heros], where Fira Math is the base math family (number 1) and XITS Math (family number 5) is used to supply other alphabets (\cal and \frak) the math parameters of the latter (XITS) are used. This can be shown in the following example:

\tracingassigns=2
\fontfam[heros]

$ a b {\cal c} $

$ \overline{a} $
\Umathoverbarvgap\textstyle=98304sp % Fira
$ \overline{a} $
\Umathoverbarvgap\textstyle=129761sp % XITS
$ \overline{a} $

\bye

This doesn't seem desirable to me. Although the LuaTeX manual clearly documents this behaviour ("7.4.2 Font-based math parameters", this link should work now: https://www.pragma-ade.com/general/manuals/luatex.pdf#1031).

Should we do something? @olsak

LuaTeX seems to always set these parameters if it detects the font as a Unicode math font, that is, it has positive number of math parameters. We could define a font feature to nil the math parameters.

[vlasakm]

Just to link this before I forget it again: https://www.tug.org/TUGboat/tb30-1/tb94vieth.pdf.

Anyways, luaotfload already has a font feature that can be used. My first rough idea:

--- a/optex/base/math-unicode.opm
+++ b/optex/base/math-unicode.opm
@@ -71,7 +71,9 @@
    \_cod -----------------------------

 \_def\_normalunimath{%
+    \_dosetmathparams
     \_loadumathfamily 1 {\_unimathfont}{} % Base font
+    \_nosetmathparams
     \_loadmathfamily  4 rsfs              % script
     \_setunimathdimens
 }%
@@ -124,7 +126,11 @@
    \_cod -----------------------------

 \_def\_umathname#1#2{"#1:\_mfontfeatures#2"}
-\_def\_mfontfeatures{mode=base;script=math;}
+\_def\_mfontfeatures{mode=base;script=math;\_mathparams}
+
+\_def\_dosetmathparams{\_glet\_mathparams=\_empty}
+\_def\_nosetmathparams{\_gdef\_mathparams{nomathparam;}}
+\_dosetmathparams

 \_def\_loadumathfamily #1 #2#3 {%
   \_font\_mF=\_umathname{#2}{\_textmff    #3} at\_sizemtext \_textfont        #1=\_mF

[olsak]

Thank you for your ideas and examples. I will do more experiments before adopting a solution.

[olsak]

Thank you, I added your ideas into master code.

[vlasakm]

Too late, but I noticed that the ConTeXt generic code also defines font feature that should do this. So no need to depend on luaotfload.

It is ignoremathconstants. I will explore it in https://github.com/olsak/OpTeX/pull/71.

[olsak]

It seems that math mathsize font feature doesn't work (Aug. 2023). It was OK in July 14. I created a tentative commit now, but maybe this is a bug in luaotfload? I don't know.

[vlasakm]

Luaotfload had an update recently. I don't think it changed anything related recently, but there maybe is a change in the ConTeXt code.

I'll be able to take a closer look in a few days.