Trailing optional args from xparse fail in array preamble

[josephwright]

For example

\listfiles
\documentclass{article}
\usepackage{xparse}
\DeclareDocumentCommand\foo{mO{xxx}}{[#1][#2]}
\usepackage{array}
\begin{document}
a $x$

\tracingmacros2

%\begin{tabular}{>{\foo{zz}[]}l}% works
\begin{tabular}{>{\foo{zz}}l}% fails
  text
\end{tabular}
\end{document}

I have a feeling this is linked to something I've seen in siunitx and therefore comes from expl3 rather than xparse.

[josephwright]

This problem reduces to the plain TeX demo

\def\foo{%
  \ifnum\iffalse{\fi`}=0 \fi
  \futurelet\testtoken\test
}
\def\test{%
  \ifnum`{=0 }\fi
}
\halign{\foo\ignorespaces#\cr test\cr}
\bye

The problem is those Appendix D tricks inside the preamble of \halign. In LaTeX2e such things only get added as required but currently in expl3 they are part of \__peek_token_generic:NNTF so get applied to all of the peek functions other than the lowest-level \peek_(g)after:Nw. (This is exactly the same issue I've had for some time in siunitx but have previously not traced it through.)

Whilst we can't be sure how alignments will work for some future LaTeX3 format, I think as a LaTeX2e package xparse does have to work here.

[blefloch]

It might actually be impossible to get this to work without changing LaTeX2e tabulars, at a minimum to include a command next to the \ignorespaces that appears in the preamble. I'll think.

[davidcarlisle]

yes or even a space character, which then gets ignored by \ignorespaces seems to be enough:

\def\foo{%
  \ifnum\iffalse{\fi`}=0 \fi
  \futurelet\testtoken\test
}
\def\test{%
  \ifnum`{=0 }\fi
}

\def\z#1{%
\halign{##\hfill&\foo\ignorespaces#1##\cr
x&  test\cr}}
\z{ }
\bye

[josephwright]

That's all very well, but does it help much? We might get some change in the LaTeX tabular mechanism to add a space there, but it's risky, and for generic mode we can do nothing.

Any idea why this happens? I took a look over Appendix D and it's not mentioned at all.

[josephwright]

BTW, probably easier than adding a space is doubling \ignorespaces! (OK, perhaps we could get away with that: it really is a no-op and could be documented for plain users.)

[josephwright]

\listfiles
\documentclass{article}
\usepackage{xparse}
\DeclareDocumentCommand\foo{mO{xxx}}{[#1][#2]}
\usepackage{array}
\makeatletter
\def\insert@column{%
   \the@toks \the \@tempcnta
   \ignorespaces\ignorespaces \@sharp \unskip
   \the@toks \the \count@ \relax}
\makeatother
\begin{document}
a $x$

%\begin{tabular}{>{\foo{zz}[]}l}% works
\begin{tabular}{>{}l}% fails
  text
\end{tabular}
\end{document}

[blefloch]

The precise problem is that TeX does not keep track of the master counter in the "u" part of an alignment preamble, but starts keeping track as soon as the last token in the "u" part is read (only reason I know that are my experiments trying to replicate TeX in TeX...). Here is an example without \futurelet.

\def\foo{\ifnum\iffalse{\fi`}=0 \fi \fooaux}
\def\fooaux#1{\ifnum`{=0 }\fi #1}
\halign{\foo XX#\cr test\cr} % works
\halign{\foo X#\cr test\cr} % fails
\halign{\foo #\cr test\cr} % works
\halign{\foo #\cr \cr} % works
\bye

We want xparse commands with trailing optional argument to look one token ahead. If we want such a look-ahead to see & as such and not as \outer endtemplate: (or as the v-part of a preamble) when these commands appear at the end of tabular cells, we need brace tricks. If we do brace tricks, then such a command will necessarily fail when it appears as the next-to-last token in the u part of an alignment preamble.

Possible resolutions:

1. remove optional arguments from xparse (not seriously)
2. remove brace tricks and accept that xparse commands cannot have &-delimited arguments (minor problem: this breaks e.g., \NewDocumentCommand{\foo}{ov}{\showtokens{#1|#2}} in \halign{#&#\cr\foo&...&\cr})
3. make sure that xparse commands with optional arguments cannot appear as the next-to-last token in an alignment preamble, by altering LaTeX2e tabular and other similar environments (problem: there are many such environments, and I don't see a clean way to hook into the primitive \halign instead to do that)
4. leave the code as it is, and document the need for \relax when using such commands in an alignment preamble (problem: this will lead to lots of bug reports downstream, as the end-user will not know that the problem has to do with xparse).

It would be interesting to check if the difficulties that Joseph had when writing siunitx macros for tables were due to the same complication.

[josephwright]

Thanks for the analysis: certainly a subtle one.

This is the same problem I have in siunitx and is not anything to do with xparse. We currently have the brace trick build-in to the \peek_... functions _except the lowest level \peek_after:Nw, which means that any look-ahead code has the same issues. (For siunitx I'm dealing with the optional argument to S-column types, and at present have to use the primitives directly.)

LaTeX2e only includes the brace trick for a specific subset of cases which are tied to \halign use. The general \@ifnextchar doesn't use it and that works in tabulars as there's always an \unskip after general cell content. The same would be true for xparse if we set it up without the brace trick.

For low-level peeking things are a bit different as that could happen in a raw \halign cell in generic mode and lead to bad things. However, I'd hope that anyone writing code using expl3 for generic use which might end up in such places should know about the brace trick and therefore would include \group_align_safe_begin:/\group_align_safe_end: as appropriate.

My feeling on this since it first came up (for me in siunitx) has been that we should go with what you've called option 2. With a proper analysis we can document exactly when (not) to use the \group_align_safe_begin:/\group_align_safe_end: pair.

(On the other options: 1 is not only a non-starter because of general usage but specifically because it rules out anything with exactly one argument which is optional, e.g. \linebreak; 3 is going to be hard to make general and doesn't help in generic mode; 4 is probably not that bad as most users won't see it but doesn't feel like a long-term fix.)

[josephwright]

Re-coding at point-of-use sorts this: for example, in siunitx, I now explicitly grab the \ignorespaces in (LaTeX) table cells before applying \peek_.... The issue then doesn't arise. At some stage we may need to document that minor restriction, but it's quite workable.