Use ConTeXt Lua code directly

[vlasakm]

Currently OpTeX uses the luaotfload package for extending LuaTeX font mechanism through Lua code.

luaotfload is a wrapper that allows use of ConTeXt code in LaTeX and plain, but as ConTeXt's luatex-plain format proves, the ConTeXt codes is actually usable directly. In fact, the bulk of the font code is available in just a single file - luatex-fonts-merged.lua.

This PR tries to experiment with using this file (and one or two other) for potential use in OpTeX.

luaotfload is not just a simple wrapper though. For OpTeX's use case not using it would have the following consequences:

• harf mode wouldn't be available (but it was never used anyways, just suggested in documentation)
• luaotfload's sophisticated mechanism for maintaining font name database couldn't be used (this includes transparent handling of system fonts)
• some special luaotfload font features wouldn't be available (colored fonts, upper/lower casing, fallback fonts, letterspace, embolden, probably more)
• I think luaotfload has some heuristics for loading bad fonts, ConTeXt just rejects them
• surely more that I am just forgetting
• different XeTeX compatibilities

But I don't see that as too much problematic:

• harf mode as of now wouldn't really be a loss
• The name database is not really needed if one uses file name not font name lookups, which I prefer, and this is also what OpTeX font family files use
• Some font features of luaotfload are duplicates of ConTeXt's, and some are really simple, so this can be fixed
• LuaTeX way is (almost?) always better than XeTeX way

As a benefit we gain:

• lose ~13000 LoC dependency that is loaded in every run from file system (spread across 29 files)
• lose another dependency - Lualibs (~10000 LoC, across ~5 files) that is required by Luaotfload, this code will still be loaded though, it just will be in the same file as the font loading code
• ability to preload font loading code into format
• I believe more things could be simplified

I am still experimenting with this, but I think it is the right way forward.

There may or may not be speed gain / loss - I think that luaotfload's database can prevent some expensive path searching, but there will be gain upfront in loading the Lua code.

I will have to check the syntax differences between luaotfload and ConTeXt's generic mode in regards to \font. But my current understanding is that generally XeTeX way of \font\a=[something] is first a lookup of font name and not file name. In the generic ConTeXt plain \font\a=something is a lookup that first tries file name and then font name (which currently doesn't work, because I don't generate any font name database).

Anyways, while there may be incompatibilities with the old way that I will have to find out, the ConTeXt way seems very nice, and maybe will allow not differentiating between "tfm" and "unicode" version of some OpTeX macros. Maybe with the code preloaded in the format, \initunifonts can even be default. And maybe we will get rid of preloaded fonts :-)

There remains the problem of distributing the two or three files.

What do you think of the idea @olsak?

[vlasakm]

Either we provide the ConTeXt code with allocator for attributes (current solution) or reserve the first 256 attributes for it.

I also found the defintion of XeTeX \font syntax, it turns out I somewhat confused " with [:

The ⟨font identifier⟩ is the only mandatory part of the above syntax. If it is given in square brackets, (e.g., \font\1="[lmroman10-regular]"), it is taken as a font file name. Without brackets, the name is looked up as both a file name and a system font name.

[vlasakm]

I was wrong even in the ConTeXt case, \font=name does font name with file name fallback. I currently let the font name lookup automatically fail, so that file names are forced.

[vlasakm]

Hm, maybe we could even \fontfam[lmfonts] by default?

With hot caches, the following (simulating \fontfam[lmfonts] in \everyjob):

\fontfam[lm]
\fontfam[heros]
test

\bye

Is:

  Time (mean ± σ):     127.0 ms ±   0.8 ms    [User: 108.8 ms, System: 17.9 ms]
  Range (min … max):   125.7 ms … 128.9 ms    23 runs

(\fontfam[lm] itself is about ~0.075.)

Compared to the situation without any \fontfam[lm], but with slow luaotfload loading:

  Time (mean ± σ):     217.1 ms ±   0.8 ms    [User: 173.9 ms, System: 42.8 ms]
  Range (min … max):   216.1 ms … 218.6 ms    13 runs

With some complexity, the opentype fonts could even be preloaded too, but I don't know about that. Now they are just read from cache. Also automatically doing \fontfam[lm] means that the fonts have to be installed and present during the run, but same situation is with the currently preloaded fonts.

[olsak]

We've talked about it before. It would be fine to leave the dependency on luaotfload, but:

• the luaotfload developers extract fontloader from Context in its actual version and add new features. They do it continuously in time. If we throw away the luaotfload then we must to do this work ourself.
• some features (like finding font by its name, no file name, XeTeX syntax) should be kept. There are many internet texts about "how to load a ned OTF font" and these instructions should work in OpTeX too (after \initunifonts, of course).
• We don't want to put OpTeX dependency to whole Context package.

[vlasakm]

We've talked about it before. It would be fine to leave the dependency on luaotfload, but:

* the luaotfload developers extract fontloader from Context in its actual version and add new features. They do it continuously in time. If we throw away the luaotfload then we must to do this work ourself.

We would need to emulate a few features if we need them. Extracting the font loader code is almost a matter of copying two files. Actually testing that the extracted stuff is working correctly is the hard part. The luaotfload people do that.

I actually think that for both testing the font loader and other OpTeX macros for regressions and other things, a test suite could be considered and users should be encouraged to contribute. Maybe a topic for "Github Discussions"?

* some features (like finding font by its name, no file name, XeTeX syntax) should be kept. There are many internet texts about "how to load a ned OTF font" and these instructions should work in OpTeX too (after \initunifonts, of course).

XeTeX syntax will work without problems.

* We don't want to put OpTeX dependency to whole Context package.

Agreed, but the files will have to be available somehow. luaotfload distrubutes them with different names. Either way the files can't be included in OpTeX due to licenses, so they will have to reside in external package.

[vlasakm]

I actually have a few ideas for the unimplemented areas of https://github.com/olsak/OpTeX/pull/67. Will see how that goes, but IIRC luaotfload doesn't allow font files with .lua extension, while the original ConTeX code does, so I thats why I will do the experiments here.

[olsak]

Either way the files can't be included in OpTeX due to licenses, so they will have to reside in external package.

This is most important issue. Suppose that we have such package, say its name is fntloader. First of all, we need a maintainer of such package. I don't feel like doing it. The maintainer must do:

• watch all news of ConTeXt's fontloader continuously.
• copy and rename relevant files from ConTeX to the fntoader.
• check all desired functionality.
• create a documentation of the fntloader package.
• upgrade documentation if there is something new.
• add a few features not included in Context's fontloader but used in OpTeX.

This job is done by luaotfload developers now. So, it seems that such maintainer will do paralell job. The benefit is that we have more simple and straightforward fntloader package than luaotfload is.

If the maintainer will stop supporting the fntloader in future (because he/she will have different interests) then OpTeX will lost the support of main package it depends on it. I see a guarantee that this will never happen with luaotfload package because there are more developers and LaTeX needs it.

[vlasakm]

Suppose that we have such package, say its name is fntloader. First of all, we need a maintainer of such package. I don't feel like doing it.

I am willing to do it. But as you say there are issues:

• watch all news of ConTeXt's fontloader continuously.

There are two possibilities:

• Keep the "Release early, release often" mentality of ConTeXt beta, and update the font code say with each release of OpTeX.
• Update the font code only once a year, from the font code:

-- Please don't update to this version without proper testing. It might be that this version
-- lags behind stock context and the only formal release takes place around tex live code
-- freeze.

• create a documentation of the fntloader package.
• upgrade documentation if there is something new.

If the code is taken more or less verbatim, ConTeXt is the documentation for better or for worse.

• add a few features not included in Context's fontloader but used in OpTeX.

Quite a lot is actually already part of ConTeXt. Others should be implemented in OpTeX. The hypothetical external package should not modify / add to the ConTeXt code in any significant way.

This job is done by luaotfload developers now. So, it seems that such maintainer will do paralell job. The benefit is that we have more simple and straightforward fntloader package than luaotfload is.

If the maintainer will stop supporting the fntloader in future (because he/she will have different interests) then OpTeX will lost the support of main package it depends on it. I see a guarantee that this will never happen with luaotfload package because there are more developers and LaTeX needs it.

Maybe this can be a joint effort of OpTeX and minim.