josephwright
commented
1 year ago The "usrguide" states:
The input given to these commands is ‘expanded’ before case changing is applied. This means that any commands within the input that convert to pure text will be case changed.
However, LICR input and literal input are handled differently regarding the Greek uppercase rules.
Indeed, that's exactly because they are expanded in a way that tries to retain as far as possible the user's choice of representation. If you try for example
\documentclass{article}
\usepackage[greek]{babel}
\makeatletter
\protected@edef\foo{\'\textalpha}\show\fooyou get \'\textalpha, which is not the same at all as ά (at least from a token-handling point of view). So the code that's set up to deal with Greek accents doesn't see the LICR version at all.
Whilst we could arrange to look for all of the combinations (\'α, etc. as well as \'\textalpha), I am not sure that is the best approach. You end up with two code paths for the same ideas, and that's asking for subtle differences. (That was the reason I moved from having separate 8-bit and Unicode handling in the first place.) We could also do a 'partial' expansion of e.g. \textalpha to α within Greek blocks, but I am worried that is error-prone.
The wider point is not just an issue for case changing. If you want to write e.g. PDF bookmarks, you need the 'pure text' equivalent of the input, which is currently handled as a separate step (and needs data which is a bit spread out). Similarly, if someone wants to map over the graphemes in some text, they'd also face the same issues. So we do need a mechanism to convert the LICRs, it's a question of where it sits.
I suspect that the best long-term fix is to adjust the 'text expansion' code such that LICRs that can be mapped to Unicode codepoints are. There would remain some issue with those that require combining chars, as they can't neatly be handled in 8-bit engines. (I suspect there an engine-dependent pathway is more-or-less inevitable.)
However, this is quite a significant change at a policy level, so I'd like wider input.
gmilde
Remaining issue from https://github.com/latex3/latex2e/issues/987.
With LICR input:
The "usrguide" states:
However, LICR input and literal input are handled differently regarding the Greek uppercase rules.
After loading babel-greek, e.g. the LICR
\'\textalphais converted to character ά (03AC GREEK SMALL LETTER ALPHA WITH TONOS).However, with xelatex or lualatex, the minimal example
results in
It seems as if the LICRs for GREEK SMALL LETTER ALPHA WITH TONOS are only partially expanded and the correct upcasing of
\acctonos\textalphais due togreek-fontenc.defwhich extends the\@uclclistwith the mapping\acctonos\LGR@hiatus(the latter prints its argument without diacritic and adds a dialytika on the second-next vowel if required for disambiguation).However, a
\@uclclistmapping of the standard accent macro\'would also affect Latin and Cyrillic characters. This is OK with 8-bit TeX, where LGR maps Latin to Greek anyway but not for Unicode fonts (TU).I experimented with a mapping
\'\accACUTE, a default to keep the accent, and Composite definitions dropping it, but did not manage to solve the problem.I wonder whether there is more detailled documentation on the working of the case-changing code. For configuration, I could imagine
\'),\DeclareUppercaseMappingthat works on a macro instead of a character, or\DeclareTextCompositeCommand).