actually looking up fonts on the filesystem

[kberry]

Hi again Will - given a TeX source with (for example) \setmonofont[Scale = .95]{zcoN-Regular.otf} I was quite surprised to learn that my .fonts.conf was relevant in any way with that specification. But it is. luaotfload still prefers system fonts to filesystem fonts, if it can match the name. Argh! (I had an old version of zcoN in my fonts.conf, and it was being found instead of the current version as found with kpsewhich.)

luaotfload-tool has an option --prefer-texmf to undo this, but I don't think that helps at the fontspec level. I asked on luaotfload, and Marcel said (https://github.com/latex3/luaotfload/issues/130) that it can be done with Lua:

\directlua{
  fonts.names.set_location_precedence { "local", "texmf", "system" }
}

wdyt? Could this be added to fontspec given the lookup-by-filename request?

He also mentioned omitting the brackets on the \font:

\font\xxx "kpse:zcoN-Regular.otf" % instead of:
%\font\xxx "[zcoN-Regular.otf]" 

though I have a feeling that would impact other things. Though I guess I don't really know. --thanks, karl.

[wspr]

Thanks -- having an option to only search kpse would make sense in fontspec. I guess it would just look like

\setmonofont{foo.otf}[OnlyKpse]

or similar. That could then change the brackets interface to a kpse: on for external filenames. Further thoughts on the interface?

[kberry]

That would be a good improvement. But I was thinking of something even stronger: make kpseonly the default behavior when filename lookups are requested. Why would I want system fonts involved at all when I explicitly asked for a filename lookup? (Answer: I don't.)

There could be an option for the reverse, [CheckSystemToo], although that doesn't sound easy to implement given the need for different syntaxes in the \font call.

However, if you think that is too incompatible a change at this point, please at least change the documentation to say that "filename lookups" aren't really -- system fonts are still found first, absent the potential new option. It was a great (and unwelcome) surprise to me.

I just hope there are no ill effects from this new kpse:foo.otf call. I'll be impressed if it all "just works" ...

[wspr]

Ah, I agree with you that Kpse only is more sensible (that’s the XeTeX approach) by default.

[zauguin]

That would be a good improvement. But I was thinking of something even stronger: make kpseonly the default behavior when filename lookups are requested. Why would I want system fonts involved at all when I explicitly asked for a filename lookup? (Answer: I don't.)

In my experience, it is quite common that people download some font, install it and then want to use it from TeX. Of course, the most obvious name is the name of the file they just downloaded (Finding the font name at the very least would involve opening the file, sometimes even finding the right tool to get such information...). So the requirement to find a system font by filename is not particularly uncommon. Actually I know multiple people who were confused by XeTeX finding different fonts when using fontnames or font filenames.

So while I think that a OnlyKpse option would be a great idea, I think that the default should stay the same.

There could be an option for the reverse, [CheckSystemToo], although that doesn't sound easy to implement given the need for different syntaxes in the \font call.

However, if you think that is too incompatible a change at this point, please at least change the documentation to say that "filename lookups" aren't really -- system fonts are still found first, absent the potential new option. It was a great (and unwelcome) surprise to me.

In which way aren't these really filename lookups? The fonts are looked up by filename, there are just more places in which we look for them.

I just hope there are no ill effects from this new kpse:foo.otf call. I'll be impressed if it all "just works" ...

The most important difference I can think of right now is that they support different characters. For obvious reasons, kpse:<name> can't be parsed correctly if <name> contains /, : or (.

[kberry]

Marcel - As far as I know, the only way fontspec provides (documents) to specify a filename lookup is by specifying a filename (foo.otf, Path=, etc.). If that syntax also does a fontname lookup, as you are saying and as you implemented in luaotfload, then there is no way to reliably do a filename lookup. So you are proposing (and have implicitly made, in luaotfload) a radical change in fontspec, it seems to me.

I fully understand that lots of people want to install and find their system fonts. I understand it is convenient to use the filenames they downloaded. Fine. But that does not obviate the need to be able to reliably find fonts by filename, without system fonts being involved.

There would be no serious problem if luaotfload preferred filename lookups to fontname lookups when a filename is requested by fontspec. But it doesn't, as you know. It finds system fonts first, even when the filename is requested. That is the crux of the problem, for me.

Furthermore, I think it is an excellent, and important, feature that xetex and luatex find the same font given the same fontspec command. This was another major point of confusion for me. I'm not sure if it is feasible for fontspec to implement in xetex the same sort of fallback to method#2 if method#1 fails that luaotfload has.

I agree that fontname vs. filename lookups have been a source of confusion ever since day one of Jonathan implementing system font lookups. And the way to improve that, it seems to me, is not to heuristically guess which one is meant, or assume "most" people use system fonts so let's use those first, but to provide clear ways to specify each separately, or both. Unfortunately Jonathan failed to provide that at the beginning (filename lookups were distinctly second-class citizens), and we are still playing catch-up, 15 years later.

I have no objection to fontspec providing a way to do filename-then-fontname, or fontname-then-filename; indeed, it sounds like a good idea. My objection is to having no way at all to do pure filename lookups (the way just about every other file in the TeX system is found).

As for what the default should be, for myself, I think the traditional fontspec method (now broken by luaotfload, as I am just learning) was reasonable: if a filename is provided, it does a filename lookup. If a fontname is provided, it does a fontname lookup. I think it would be a terrible incompatibility to change that, but if that's what Will decides, that's what he decides. Given a [KpseOnly] or [SystemFontOnly] or [TryBoth] options, it would be at least be possible for people to get the behavior they want.

I didn't know kpse:name did not allow . That makes it less valuable. That's another common thing people want to do: give an absolute path to the exact file they want. Kpse itself certainly supports absolute path lookups. Oh well. For "normal" filename lookups controlled by the paths, it would still be a good improvement to be able to access that with a TeX option, as discussed. -k

[davidcarlisle]

On Tue, 11 Feb 2020 at 23:44, kberry notifications@github.com wrote:

I have no objection to fontspec providing a way to do filename-then-fontname, or fontname-then-filename; indeed, it sounds like a good idea. My objection is to having no way at all to do pure filename lookups (the way just about every other file in the TeX system is found).

Karl I find your terminology confusing here,

A filename lookup uses the system filename of the font file a fontname lookuop uses the internal font name as specified inside the font.

That distinction is separate from the distinction of whether the system font path or the texlive font search path is used, isn't it?

David

[kberry]

Karl I find your terminology confusing here,

I find your confusion confusing!

A filename lookup uses the system filename of the font file

I don't know what the "system filename of the font file" is, but no, that is not my understanding.

a fontname lookuop uses the internal font name as specified inside the font.

I guess. The reality, as far as I've seen, is that a "fontname" lookup uses fontconfig (on Unix). No one seems to know what that really means or how to control it precisely. I have tried pretty hard to understand why fontconfig returns font x for request y, and failed. There is no clear documentation. The source is unreadable. Etc.

That distinction is separate from the distinction of whether the system
font path 

System fonts (on Unix anyway) do not exactly have a "path". They are in some weird database with all kinds of tricky features.

or the texlive font search path is used, isn't it?

This is all news to me. My understanding of fontspec (and xetex) ever since day one was that a filename request (foo.otf) meant looking it up with kpse, as in "kpsewhich foo.otf", and did not involve system fonts in any way whatsoever. That is certainly what the documentation implies. It does not talk about the "system filename of the font file".

Hans implemented all kinds of crazy caching of system fonts in luatex which obscured (and still obscures, I guess) the issue. I hated this but have no influence over what hapens.

This is all so frustrating!!!

[zauguin]

Karl I find your terminology confusing here,

I find your confusion confusing!

A filename lookup uses the system filename of the font file

I don't know what the "system filename of the font file" is, but no, that is not my understanding.

The "system filename of the font file" is the name of the file in which the font is stored.

a fontname lookuop uses the internal font name as specified inside the font.

I guess. The reality, as far as I've seen, is that a "fontname" lookup uses fontconfig (on Unix). No one seems to know what that really means or how to control it precisely. I have tried pretty hard to understand why fontconfig returns font x for request y, and failed. There is no clear documentation. The source is unreadable. Etc.

You can find the logic used by luaotfload in luaotfload-database.lua.It is a bit chaotic but with some effort readable and tries to be mostly compatible with fontconfig. Basically what it does is: Every font has a fontname, a (preferred) familyname and additional metadata set (in the OpenType name table). When a name is given, it is matched against the familynames. If a familyname matches, a font from that family is used (The precise font depends on whether regular/italic/bold has been requested, optical sizes etc). Otherwise we match against the fontname. If a fontname matches, the matching font is used. (fontconfig allows some additional lookup variants which are mostly irrelevant in a TeX context)

That distinction is separate from the distinction of whether the system font path

System fonts (on Unix anyway) do not exactly have a "path". They are in some weird database with all kinds of tricky features.

??? If there are fonts on Unix stored in some weird database then luaotfload can't find them. As far as we are concerned, the fontconfig configuration files translate to a list of directories (mostly /usr/share/fonts on a typical system). System fonts are the font files found in this directory. (fontconfig also has some index files, but they are more of an implementation detail.) The file names of the files in this directories are the filenames which are relevant when doing a filename lookup in luaotfload.

or the texlive font search path is used, isn't it?

This is all news to me. My understanding of fontspec (and xetex) ever since day one was that a filename request (foo.otf) meant looking it up with kpse, as in "kpsewhich foo.otf", and did not involve system fonts in any way whatsoever. That is certainly what the documentation implies. It does not talk about the "system filename of the font file".

What David describes is pretty much the model of luaotfload (XeTeX is a bit different): There are two distinctions:

• Lookups by filename: A file with the requested name is requested. The font file with the requested name can be either in the system font path or in texmf. This also corresponds to the documentation of fontspec:

  When selecting fonts by file name, any font that can be found in the default search paths may be used directly

  Of course the question is what default search paths are, but as far as fonts are concerned I would definitly describe the default system font path as default search paths.

• Lookups by fontname: A font with the requested familyname or fontname (as described above) is requested. Again, this can find system fonts or fonts in texmf.

Hans implemented all kinds of crazy caching of system fonts in luatex which obscured (and still obscures, I guess) the issue. I hated this but have no influence over what hapens.

All relevant caching happens in luaotfload, luatex (the engine) itself never deals with system font related stuff. This caching is pretty much necessary to get performant font name lookup because we can't use fontconfig directly and they don't have a documented (and there probably not stable) format for their cache files, so we can't reuse them either.

[davidcarlisle]

On Wed, 12 Feb 2020 at 00:07, kberry notifications@github.com wrote:

Karl I find your terminology confusing here,

I find your confusion confusing!

A filename lookup uses the system filename of the font file

I don't know what the "system filename of the font file" is, but no, that is not my understanding.

I meant just the filename of the file, as returned by ls (or dir or whatever) not using any special properties of the file being a font.

As opposed to the internal name(s) set in the internal tables of the font structure in the file.

David

[kberry]

Marcel - thanks for all the information. I forgot that luatex does not use fontconfig, only xetex does, so all that was off the mark. Sorry.

dc> I meant just the filename of the file, as returned by ls (or dir or
whatever) not using any special properties of the file being a font.

Yes, right. But ... as found by kpse, using the standard TeX-world paths, not involving system font paths (like /usr/share/fonts) or fontconfig (xetex). That is what I had always understood "filename lookup" to mean, as far as I understood it, and how the fontspec documentation talks about it. Thus it was news to me that luaotfload looks in the system font directories even when a filename is requested.

For myself, I think this difference between xetex and luatex is undesirable. If it is kept as the default by fontspec, at least it should be documented and the [KpseOnly] option come back into play, so those of who need reproducible, system-independent documents can have them. We'll see what Will thinks (who mentioned to me he's busy at work so nothing will happen soon, anyway).

As opposed to the internal name(s) set in the internal tables of the font
structure in the file.

That's what I thought you meant. I'm glad you didn't :). I was at my wit's end yesterday. --thanks, karl.

[wspr]

Thanks all for the discussion. I have to admit I’m leaning towards keeping the current behaviour simply for backwards compatibility, and allowing [OnlyKpse] to be a font option so you could write \defaultfontfeatures{OnlyKpse} in a fontspec.cfg file and change the behaviour globally for oneself. Anyone else like to weigh in on the relative merits of either approach?