Handling of & in memoized code

[cfr42]

This may be a documentation request because it isn't clear to me that memoize could avoid errors here. But it would be good if memoize could either identify the problem (preferably) or include an explanation in the manual. If the issue is documented and I didn't find it, indexing &, ampersand and/or the error and/or the most common environment/package triggers would be helpful.

MNWE

\documentclass{article}
\usepackage{memoize}
\usepackage{tikz}
\usetikzlibrary{matrix}
\begin{document}
\begin{tikzpicture}
  \matrix (m) [matrix of nodes]
  {
    a & b & c\\
    d & e & f\\
  };
\end{tikzpicture}
\end{document}

This fails as follows

(./prawf.944AD362A4B16C60FEC388997EA807BE.memo)
! Undefined control sequence.
<argument> \pgf@matrix@last@nextcell@options

l.893 \end{tikzpicture}

?
! Undefined control sequence.
<argument> \pgf@matrix@last@nextcell@options

l.893 \end{tikzpicture}

?
Missing character: There is no b in font nullfont!

! Package pgf Error: Single ampersand used with wrong catcode.

See the pgf package documentation for explanation.
Type  H <return>  for immediate help.
...

l.893 \end{tikzpicture}

?

Only the last error gives any clear indication of what's wrong. Since most editors will stop at the first error, many users won't get that hint.

It would be helpful to document the(?) solution.

\begin{tikzpicture}[ampersand replacement=\&]
  \matrix (m) [matrix of nodes]
  {
    a \& b \& c\\
    d \& e \& f\\
  };
\end{tikzpicture}

Perhaps also worth noting that forest doesn't require this.

\documentclass{article}
\usepackage{memoize}
\usepackage{forest}
\begin{document}
\begin{forest}
  for tree={align=cc},
  [a
    [b & c]
  ]
\end{forest}
\end{document}

This case works as-is, presumably because forest handles the catcodes or memoize includes special handling.

Note that I've still got version 1.2, but I looked through the change log and searched the new documentation for ampersand and didn't see any relevant changes. I've also read most of the issues on github and don't remember this coming up.

[sasozivanovic]

Indeed, I cannot address this other than in the documentation.

The issue is not the ampersand per se. Any environment which changes the category code of some character (TikZ matrices change the catcode of & to 13, active) will run into the same problem, and receive — from Memoize — the same solution: verbatim. (Incidentally, your Forest example does not present a problem because Forest's align uses the standard tabular environment, which does not change the catcode of &.)

As far as I see, the best I can do is to emphasize the category codes in the verbatim section of the tuturial (2.9). As it stands, the section might give the impression that key verbatim is only useful for verbatim environments, whereas in fact it presents the solution to all catcode problems. I actually like the matrix example a lot; I think I want to include it in this section. This section might also be the best place for the suggested list of the most common environments exhibiting the problem.

I will also add the index entry, as suggested — something like "category codes, problems with" — and a paragraph to the troubleshooting section (6.3).

[cfr42]

Thanks. I figured it was most likely a doc request.

Do I understand correctly that the problem arises from the fact that an environment makes the character active? So if & was active throughout the document, it wouldn't be a problem? To be clear: I do realise this would cause other problems. I'm not asking if this would 'solve' anything; I'm just trying to understand what happens.

memoize doesn't seem to care whether I use verbatim or not for characters I've made active generally.

I suspect people are unlikely to find the index entry if they don't see the second part of the error message, which they won't if their editor just halts immediately on error. So I do think including some common examples of troublesome environments in the troubleshooting would be really helpful. Even if people run into the problem for a different environment, at least some of them will pick up the similarity.

(Incidentally, your Forest example does not present a problem because Forest's align uses the standard tabular environment, which does not change the catcode of &.)

Sorry. I should disappear.

[sasozivanovic]

Do I understand correctly that the problem arises from the fact that an environment makes the character active? So if & was active throughout the document, it wouldn't be a problem?

Yes, I believe this is the case.

I suspect people are unlikely to find the index entry if they don't see the second part of the error message, which they won't if their editor just halts immediately on error.

I am being too nasty if I suggest they get another editor then? ;-)

I mean, in the (La)TeX world, error messages are cryptic enough as it is, and often enough, the first error message is not the most informative ...

So I do think including some common examples of troublesome environments in the troubleshooting would be really helpful. Even if people run into the problem for a different environment, at least some of them will pick up the similarity.

I completely agree!

(Incidentally, your Forest example does not present a problem because Forest's align uses the standard tabular environment, which does not change the catcode of &.)

Sorry. I should disappear.

This, I absolutely forbid!

[sasozivanovic]

Btw, as you can see in issue #19, I'm now considering a major revision of the verbatim section of the tutorial.

[cfr42]

I suspect people are unlikely to find the index entry if they don't see the second part of the error message, which they won't if their editor just halts immediately on error.

I am being too nasty if I suggest they get another editor then? ;-)

I mean, in the (La)TeX world, error messages are cryptic enough as it is, and often enough, the first error message is not the most informative ...

I'm ashamed to admit that every editor I've tried or used has done this except for Overleaf. (Actually my editor has recently gone somewhat mad, so hiding compilation errors is not high on my list of complaints.)

[cfr42]

This, I absolutely forbid!

Thank you.

[sasozivanovic]

That's good to know, I only ever used Emacs + AUCTeX ... (and Overleaf, occasionally).

[sasozivanovic]

I pushed the documentation update of section 2.9 discussed above. See prerelease 1.4.0-wip.

[sasozivanovic]

I also created a Wiki page, here at GitHub, which lists the issues caused by various commands (and provides a solution, if possible). The idea is that this list would be a community effort ;-)

[sasozivanovic]

(I'm experimenting with the form of the list, let me know what you think.)

[cfr42]

I like the structure and I think this could be very useful, but the use of auto is a bit confusing. On the one hand, it refers to things you have to do to automatise memoization. On the other, it refers to things you don't have to do to automatise memoization. That makes perfect sense in the docs, but here it seems a bit odd to list something with label auto when that's described as a label for issues concerning automatisation. I think a different label for one or the other would be more useful from a user perspective. Or maybe auto versus auto (config) or something? So it's clear it is the same process but obvious at a glance whether anything needs to be done to enable that process?

Tikz:matrix -> Tikz as I assume the issue arises whether the library is used or not. Or does Tikz:matrix mean something else here?

Hyperlinks would be a useful addition?

I'm not sure about the 'community' bit. There's no obvious way to suggest changes? Or am I supposed to clone it and submit a pull request? I worry this will be just another thing you have to do.