dealing with custom tokens

[kimbauters]

What I want to achieve is output similar to the following: The image shows the HTML output and shows that everything is indeed working. To accomplish this, I created a new token (balloon) in Pygments and subclass all the languages for which I want to support this balloon annotation, ensuring this token is parsed first.

When I use this in LaTeX through minted, it doesn't quite seem to work though. As far as I can tell minted does pick up on the modifications to Pygments (the control symbols to signal a balloon are correctly removed) but the LaTeX output doesn't match.

What I get through pygmentize is

\begin{Verbatim}[commandchars=\\\{\}]
\PY{k+kd}{func} \PY{balloon}{function name}\PY{p}{(}\PY{balloon}{argument} \PY{balloon}{parameter}\PY{p}{:} \PY{balloon}{type}\PY{p}{,} \PY{p}{...)} \PY{p}{\PYZob{}}
  \PY{balloon}{statements}
\PYG{p}{\PYGZcb{}}
\end{Verbatim}

The output of minted, however, is

\begin{Verbatim}[commandchars=\\\{\},codes={\catcode`\$=3\catcode`\^=7\catcode`\_=8}]
\PYG{k+kd}{func} \PYG{n+nf}{function} \PYG{n}{name}\PYG{p}{(}\PYG{n}{argument} \PYG{n}{parameter}\PYG{p}{:} \PYG{n}{type}\PYG{p}{,} \PYG{p}{.}\PYG{p}{.}\PYG{p}{.}\PYG{p}{)} \PYG{p}{\PYGZob{}}
  \PYG{n}{statements}
\PYG{p}{\PYGZcb{}}
\end{Verbatim}

So it seems like extra processing is taking place which is removing my newly defined token.

How can I ensure this newly created token survives the processing by minted? I tried to look through the code for minted, but LaTeX packages always tend to overwhelm me. Also, any idea how I can achieve this particular styling afterwards? My default solutions would be mdframed or tikz, but I'm not quite sure how those would work here.

[gpoore]

minted is just calling pygmetize. There isn't any extra processing. You might check to make sure that the right lexer is being specified in the LaTeX document, and also see whether you have multiple pygmentize installations. Deleting the minted cache (directory starting with _minted) would also help, if you haven't tried that already.

In terms of styling, you probably do want something like TikZ. You'll probably want to define your own custom style macros. This might give you some ideas about one way to do that, if you don't want to do the modifications in Pygments itself: https://tex.stackexchange.com/a/232760/10742.

[kimbauters]

Thanks for that. The right lexer is indeed being specified. The input is something akin to ßfunction nameß where the output given through minted is \PYG{n+nf}{function} PYG{n}{name}. So the lexing is clearly happening as the ß are gone. The correct pygmentize is also called. It’s a symbolic link to where Pygments is installed, and renaming it causes minted to throw errors.

Could you tell me exactly how pygmetize is called (I’m wondering how the \PY ended up as \PYG) and/or where to look into the code as to where the call is made? Also, where is the styling information coming from? Is this from the Python files in the Pygments styling folder?

I will try the _minted folder tomorrow as well as the stackoverflow suggestion. Thanks for that link!

update: it's not the _minted folders, those are typically deleted in between typesetting.

[kimbauters]

A bit of an update on trying to find out what is happening. The only problem that remains at this moment is the output generated by minted.

When I switch back from my custom lexer to the default Swift lexer, both minted as well as a command line invocation give the same output:

\begin{Verbatim}[commandchars=\\\{\},codes={\catcode`\$=3\catcode`\^=7\catcode`\_=8}]
\PYG{k+kd}{func} \PYG{err}{§}\PYG{n}{function} \PYG{n}{name}\PYG{err}{§}\PYG{p}{(}\PYG{err}{§}\PYG{n}{argument}\PYG{err}{§} \PYG{err}{§}\PYG{n}{parameter}\PYG{err}{§}\PYG{p}{:} \PYG{err}{§}\PYG{n}{type}\PYG{err}{§}\PYG{p}{,} \PYG{p}{.}\PYG{p}{.}\PYG{p}{.}\PYG{p}{)} \PYG{p}{\PYGZob{}}
\end{Verbatim}

When I switch back to my own lexer (with or without modifying the python file for the style I'm using to accommodate the extra token) the output of the command line invocation is as expected:

\begin{Verbatim}[commandchars=\\\{\}]
\PYG{k+kd}{func} \PYG{balloon}{function name}\PYG{p}{(}\PYG{balloon}{argument} \PYG{balloon}{parameter}\PYG{p}{:} \PYG{balloon}{type}\PYG{p}{,} \PYG{p}{.}\PYG{p}{.}\PYG{p}{.}\PYG{p}{)} \PYG{p}{\PYGZob{}}
\end{Verbatim}

while the output of minted shows partial processing (the err tags are gone) as well of signs of additional or different processing (the function name, for example, is split up in two distinct tags):

\begin{Verbatim}[commandchars=\\\{\},codes={\catcode`\$=3\catcode`\^=7\catcode`\_=8}]
\PYG{k+kd}{func} \PYG{n+nf}{function} \PYG{n}{name}\PYG{p}{(}\PYG{n}{argument} \PYG{n}{parameter}\PYG{p}{:} \PYG{n}{type}\PYG{p}{,} \PYG{p}{.}\PYG{p}{.}\PYG{p}{.}\PYG{p}{)} \PYG{p}{\PYGZob{}}
\end{Verbatim}

I must be missing something, but I can't figure out what it is.

[gpoore]

Have you tried deleting everything in the _minted-<filename> directory? By default, minted won't update anything as long as those cache files exist. Can you post the complete .tex source you are compiling?

[kimbauters]

I have identified where the issue lies. It has to do with the escapeinside option that I use, and which is passed on to Pygments. When removing that option, the LaTeX code with minted typesets exactly as it should. As for the .tex, it is quite simple so there wasn't much else to try:

\documentclass{article}
\usepackage{minted}               % enable code highlighting
\usemintedstyle{friendly}

\begin{document}
\section*{}
\begin{minted}[]{swift}
func §function name§(§argument§ §parameter§: §type§, ...) {
\end{minted}

\end{document}

edit: all fixed. I had to modify the Pygments LaTeX formatter to ignore these balloons just as it ignores strings and comments. Thanks for your comments though, as they definitely helped me pinpoint where the issue was. And thanks for making and maintaining the minted package, of course :)