blefloch / latex-unravel

Watching TeX digest tokens
25 stars 1 forks source link

Temp token `\notexpanded:<token>` is not properly restored #57

Open muzimuzhi opened 1 year ago

muzimuzhi commented 1 year ago
\documentclass{article}
\usepackage{unravel}

\unravel{\expandafter\relax\noexpand\cmd}

\begin{document}
\leavevmode\ttfamily
\unravel{\detokenize\expandafter{\noexpand\cmd}}
\end{document}
||
|> \expandafter \relax \noexpand \cmd

[===== Step 1 =====] \expandafter \relax
|| \expandafter \relax
|> \noexpand \cmd

[===== Step 2 =====] back_input: \expandafter \relax
||
|> \relax \notexpanded:\cmd

The second example \unravel{\detokenize\expandafter{\noexpand\cmd}} also gives wrong output.

muzimuzhi commented 1 year ago

I now think current behavior is the expected one. TeX-SX question https://tex.stackexchange.com/q/12482 has helped me a lot in realizing this.

If <token> is expandable, then \noexpand<token> expands to a temporary token \notexpanded:<token> whose meaning is equal to \relax.

muzimuzhi commented 1 year ago

Nope, this only explains the fist example. For the second example, the different outputs with or without outer \unravel are clearly a problem.

Another case:

\documentclass{article}
\usepackage{unravel}

\begin{document}
\ttfamily

\def\mycmd#1{\detokenize{#1}}
         \expandafter\mycmd\noexpand\cmd\par
\unravel{\expandafter\mycmd\noexpand\cmd}
% Expected: "\cmd "
% Actual:   "\notexpanded:\cmd "

         \expandafter\meaning\noexpand\cmd\par
\unravel{\expandafter\meaning\noexpand\cmd}
% Expected: "\relax"
% Actual:   "undefined"
\end{document}
blefloch commented 9 months ago

That's a bug, but difficult to fix as the expansion-preventing aspects of \noexpand are very temporary and a true "\notexpanded:\foo" token cannot be stored. Instead I have cheated by introducing actual control sequences with that name. But this leads to the issue here.