Inconsistent syntax

[Udi-Fogiel]

Currently the source contains several methods for various tasks, e.g. for defining commands there are TeX primitives (\def, \let etc.), LaTeX2e commands (e.g. \newcommand), LaTeX3 commands (e.g. \NewDocumentCommand), expl3 functions (e.g. \cs_new:n) and commands from the makecmds package (e.g. \provideenvironment). I think the source would be much more readable if it was more consistent, with well structured syntax rules.

There are several aspects to consider. The first, is what methods to use. For example, the following essentially do the same https://github.com/reutenauer/polyglossia/blob/3737629d973c1b197b9c0342df6877555c4a3954/tex/polyglossia.sty#L2771-L2775 https://github.com/reutenauer/polyglossia/blob/3737629d973c1b197b9c0342df6877555c4a3954/tex/polyglossia.sty#L1436-L1437 but in a first glance they look rather different.

Another aspect is macros (or functions) names. The source have two prefixes for internal commands, xpg and polyglossia. I'm not sure what is the difference, nor what is the meaning of xpg. It does not seem like the naming conventions specified in interface3 (section 1.1) are respected (using @ for function names, using \DeclareDocumentCommand for private functions) and scratch variable syntax can be improved.

IMO the best approach would be to rewrite everything in expl3 and LaTeX3 syntax, and to use l3keys2e only (not xkeyval). This is because most of the code is already dependent on it, and it offers more functionality than the other options. But, the most important thing is that the person who deals with the code most will understand it.

It probably should be done is several stages, while testing after each stage.

[jspitz]

I have also started to transform things to LaTeX3 or modern LaTeX syntax. But it is quite some work and needs to be tested carefully. We also need to consider with some internal macros that other packages might patch them (that's why I refrained from unifying the xpg and polyglossia namespace until now; xpg is the old one, I think it means "external package polyglossia" or maybe "XeteX polyglossia" given that this was the original scope or something along these lines. Of course polyglossia is more transparent, but xpg is more economic and quite a neat acronym, so I tend to go for that at least for internal pseudo-namespace)

Another thing on my agenda is getting rid of xkeyval and using l3keys[2e] instead, but we can only do that once for all gloss files.

[Udi-Fogiel]

We also need to consider with some internal macros that other packages might patch them (that's why I refrained from unifying the xpg and polyglossia namespace until now;

Hmm I'm not sure what to do about that other than make a public statement. But this can wait, we can start with dealing with the private internal functinos, and removing unnecessary dependencies (such as expl3 package, or luatexbase).

[Udi-Fogiel]

Another thing on my agenda is getting rid of xkeyval and using l3keys[2e] instead, but we can only do that once for all gloss files.

Does this mean that gloss files will eventually use expl3 syntax?

[jspitz]

I would rather avoid a general rewrite of the gloss files. They need to be accessible to their respective maintainers. I think the LaTeX2e l3keys interface will do for most, even though it doesn't provide for choice keys yet.

Note that one gloss (Latin) already employs expl3 syntax (Keno Wehr's choice).

[Udi-Fogiel]

I would rather avoid a general rewrite of the gloss files. They need to be accessible to their respective maintainers.

sure, I agree.

I think the LaTeX2e l3keys interface will do for most, even though it doesn't provide for choice keys yet.

Is there one? how can I define a new key in "normal"catcode regime?

[jspitz]

Is there one? how can I define a new key in "normal"catcode regime?

\DeclareKeys[⟨family⟩]{⟨declarations⟩}

(and \ProcessKeyOptions, \SetKeys etc.)

See clsguide, sec. 4.4

This has been introduced in LaTeX release 2022-06-01

[Udi-Fogiel]

Oh, this is great! I did not know about this.

[jspitz]

Yes, it's pretty neat. I hope they'll extend the syntax,

[Udi-Fogiel]

what is the reason for patching \XKV@define@default? will we still need something similar?

[jspitz]

what is the reason for patching \XKV@define@default?

See #364 and f71fc8166f12df7c8f

will we still need something similar?

We need a way to access the default value associated with a key (in case no value is given). On a quick glance I haven't found something useful in l3keys.

[Udi-Fogiel]

I don't think there is a notion of 'default value' in \DeclareKeys. Maybe we can define our own parser. see section 27.10 of interface3.

[jspitz]

l3keys has .default:n to set a default.

[Udi-Fogiel]

yes, but I thought we are trying to not enforce gloss files to use L3 interface.

[jspitz]

I have never tried actually, but clsguide states that

A small number of ‘basic’ ⟨properties⟩ are described below.
The full range of properties, provided by l3keys, can also be used for more
powerful processing. See interface3 for the full details.

I am not sure whether this means that the l3keys properties can be passed as is to \DeclareKeys, but it looks so (we just need to try).

[Udi-Fogiel]

works!

[Udi-Fogiel]

works for choice keys as well

[Udi-Fogiel]

We need a way to access the default value associated with a key (in case no value is given). On a quick glance I haven't found something useful in l3keys.

Here is an example to explain how we can do that

\documentclass{article}

\DeclareKeys[foo]{bar.code:n = #1, bar.default:n = baz}

\begin{document}
\csname\detokenize{key default > foo/bar}\endcsname

\end{document}

[Udi-Fogiel]

@jspitz I'm working on a transition to l3keys, but I don't understand something. Why the difference in this example:

\documentclass{article}
\usepackage{polyglossia}
\setmainlanguage{french}
\begin{document}
\selectlanguage[variant=canadian]{french}

\babelname

\selectlanguage{french}

\babelname

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

a?

\selectlanguage[autospacing=false]{french}
a?

\selectlanguage{french}
a?

\end{document}

i.e. why the variant is restored but the autospacing is not? This also implies that if I will use an alias with autospacing=false it will produce unexpected result

\documentclass{article}
\usepackage{polyglossia}
\setmainlanguage{french}
\setlanguagealias[autospacing=false]{french}{foo}
\begin{document}

a?

\selectlanguage{foo}
a?

\selectlanguage{french}
a?

\end{document}

To me, it looks like a bug, but I'm not sure. I always though that the options selected at language loading time are the default ones for this language.

[jspitz]

I am not sure. With the language environments and inline commands (\textfrench), options are local. With \selectlanguage as well as \foreignlanguage and maybe others current options are inherited.

I am not sure whether this is explicit design, and I am also not sure whether I like it or not. But it is established behavior, and we will interfere with people's documents if we change this now (the solution would probably be as easy as clearing the option list and resetting defaults at the beginning of \xpg_alias_add_to_option_i:nn).

[Udi-Fogiel]

The documentation says otherwise:

The command \foreinlanguage{⟨lang⟩}{⟨…⟩} and the otherlanguage* environment are identical with the use of the \text⟨lang⟩ or \textlang command, with the one notable exception that they do not translate the date with \today.

[jspitz]

Yes, but it behaved differently.

[Udi-Fogiel]

Then to me it is clearly a bug :) In any case, can you point to where variants are stored and recovered?, as they are clearly an exception here.

[jspitz]

\__xpg_set_language_options:nnn

[Udi-Fogiel]

is it stored in \<main/other/>langvariant? this is the only difference I see from other keys.

[Udi-Fogiel]

To be more precise, how exactly the variant is restored here?

\documentclass{article}
\usepackage{polyglossia}
\setmainlanguage[variant=canadian]{french}
\begin{document}

\babelname

\selectlanguage[variant=acadian]{french}

\babelname

\selectlanguage{french}

\babelname

\end{document}

I don't see how the variant key is treated differently in \__xpg_set_language_options:nnn other than storing the variant in a macro for external packages.

[Udi-Fogiel]

Ok, it probably isn't, this is just a bug in \babelname...

\documentclass{article}
\usepackage{polyglossia}
\setmainlanguage[variant=canadian]{french}
\begin{document}

\languagevariant

\babelname

\selectlanguage[variant=acadian]{french}
\languagevariant

\babelname

\selectlanguage{french}
\languagevariant

\babelname

\end{document}

[Udi-Fogiel]

Or maybe it is restored?

\documentclass{article}
\usepackage{polyglossia}
\setmainlanguage[variant=canadian]{french}
\begin{document}

\languagevariant

\babelname

\BCPdata{tag}

\selectlanguage[variant=acadian]{french}
\languagevariant

\babelname

\BCPdata{tag}

\selectlanguage{french}
\languagevariant

\babelname

\BCPdata{tag}

\end{document}

but then \languagevariant is bugged.

[jspitz]

Both \languagevariant and \babelname are not correctly set on \selectlanguage

[Udi-Fogiel]

and \BCPdata?

[jspitz]

and \BCPdata?

also :-(

[Udi-Fogiel]

I think we should consider changing the behavior of \foreignlanguage and \selectlanguage...

[jspitz]

This is a bug in the repo which isn't in the last release. I'll check.

[Udi-Fogiel]

And it looks like \foreignlanguage changes the parameters locally.

\documentclass{article}
\usepackage{polyglossia}
\setmainlanguage{hebrew}
\setmainfont{David CLM}
\begin{document}
\foreignlanguage[numerals=hebrew]{hebrew}{\localnumeral{12}}

\foreignlanguage{hebrew}{\localnumeral{12}}
\end{document}

This is a bug in the repo which isn't in the last release. I'll check.

Oh, I should have mentioned, I tested everything with v.165, so I'm not sure what is the correct result anymore. I'll update my prevoius comments with pictures.

[jspitz]

git bisect points out 55de651eeba as the commit that introduced the bug

[Udi-Fogiel]

Again, I'm not sure how you deduced that because I don't have an example with expected result, this picture is obviously wrong and produced with v1.65... If you have a good example, Ill be hapy to fix that.

[Udi-Fogiel]

And if the \babelname in the linked picture is correct, how come it gets restored, and all other information related to the variant, but other keys are inherited?

[jspitz]

Take this:

% !TeX TS-program = xelatex
\documentclass{article}
\usepackage{polyglossia}
\setmainlanguage[variant=canadian]{french}
\setotherlanguage{german}
\begin{document}

\section{Canadian French}
    Variant: \languagevariant

    Babelname: \babelname

    BCP-47: \BCPdata{tag}

\section{Acadian French}
    \selectlanguage[variant=acadian]{french}
    Variant: \languagevariant

    Babelname: \babelname

    BCP-47: \BCPdata{tag}

\section{No Option French (= Canadian)}
    \selectlanguage{french}
    Variant: \languagevariant

    Babelname: \babelname

    BCP-47: \BCPdata{tag}

\section{Austrian German}
    \selectlanguage[variant=austrian]{german}
    Variant: \languagevariant

    Babelname: \babelname

    BCP-47: \BCPdata{tag}

\end{document}

With 1.66:

With current master:

As you can see, the babelname and BCP47 value is wring, variant is correct.

[jspitz]

More correctly, variant is wrong in both cases for the option-less switch (sec. 3)

[jspitz]

git bisect points out 55de651 as the commit that introduced the bug

I have fixed this at fe72d6405db416f52f

[Udi-Fogiel]

I have fixed this at fe72d64

was just about to push the same. Thanks!

[Udi-Fogiel]

It still does not explain why all the information regarding the variant key is restored but all other keys are not.

[Udi-Fogiel]

And it looks like \foreignlanguage changes the parameters locally.

\documentclass{article}
\usepackage{polyglossia}
\setmainlanguage{hebrew}
\setmainfont{David CLM}
\begin{document}
\foreignlanguage[numerals=hebrew]{hebrew}{\localnumeral{12}}

\foreignlanguage{hebrew}{\localnumeral{12}}
\end{document}

Never mind, now I understand what you meant.

[Udi-Fogiel]

Take this:

% !TeX TS-program = xelatex
\documentclass{article}
\usepackage{polyglossia}
\setmainlanguage[variant=canadian]{french}
\setotherlanguage{german}
\begin{document}

\section{Canadian French}
  Variant: \languagevariant

  Babelname: \babelname

  BCP-47: \BCPdata{tag}

\section{Acadian French}
  \selectlanguage[variant=acadian]{french}
  Variant: \languagevariant

  Babelname: \babelname

  BCP-47: \BCPdata{tag}

\section{No Option French (= Canadian)}
  \selectlanguage{french}
  Variant: \languagevariant

  Babelname: \babelname

  BCP-47: \BCPdata{tag}

\section{Austrian German}
  \selectlanguage[variant=austrian]{german}
  Variant: \languagevariant

  Babelname: \babelname

  BCP-47: \BCPdata{tag}

\end{document}

Can you explain why in \section{No Option French (= Canadian)} the result of \babelname is canadien? From

With \selectlanguage as well as \foreignlanguage and maybe others current options are inherited.

I expected the variant to be inherited as well...

[jspitz]

The variant inheritance is fixed at 54000b1e0b40c

[Udi-Fogiel]

I'm taking about the key, not the command \languagevariant. Let's go back to my first example

\documentclass{article}
\usepackage{polyglossia}
\setmainlanguage{french}
\begin{document}
\selectlanguage[variant=canadian]{french}

\babelname

\selectlanguage{french}

\babelname

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

a?

\selectlanguage[autospacing=false]{french}
a?

\selectlanguage{french}
a?

\end{document}

Do you see that autospacing's value is inherited, but the variant's is not?

[jspitz]

But you agree that what I fixed was a bug as well? Yes, I see that autospacing should be reset to default here, but isn't.

[jspitz]

I suppose we need to store the keys and values in \__xpg_load_lang_definition:nn and restore them in case a language is switched to without options.

[jspitz]

We do save the current options of a language in xpg@current@options@<lang>. If we'd do the same for the initial setting of a language, we could restore that.

[jspitz]

(see for instance the otherlanguage environment, where we restore the pre-switch options after the environment has been closed)

[Udi-Fogiel]

But you agree that what I fixed was a bug as well?

Of course!

Yes, I see that autospacing should be reset to default here, but isn't.

Ok, that has answered my initial question. so what did you mean by the following?

With \selectlanguage as well as \foreignlanguage and maybe others current options are inherited.

that these inherit the initial keys?