[bug] `gobble` results in malformed formatting, `autogobble` works

[goyalyashpal]

MWE:

Note: here

• the 4 spaces are caused due to indentation inside the filecontents block,
• so, autogobble and gobble=4 shall be equivalent,
• as can be seen from eyeballing too, that both the outputs align
• but something else too is happening in gobble (like say maybe insertion of some ghost character or change of encoding etc) which causes the pygments to fail to recognize it as proper file & hence results in malformed formatting
• compiled with xelatex --shell-escape document.tex

\begin{filecontents}[noheader, overwrite]{./test.mysql}
    mysql-> select * from firstLine;
    mysql-> select * from secondLine;
    mysql-> \help contents
    mysql-> \h contents
\end{filecontents}

\documentclass{article}
\usepackage{minted}

\begin{document}

\inputminted[autogobble]{psql}{test.mysql}    % Works
\inputminted[gobble=4]{psql}{test.mysql}    % Buggy

\end{document}

---

just for my reference, first noticed filecontents at https://github.com/gpoore/minted/issues/378#issuecomment-1794025908

[muzimuzhi]

Seems a pygments bug.

$ cat test.mysql
    mysql-> select * from firstLine;
    mysql-> select * from secondLine;
    mysql-> \help contents
    mysql-> \h contents

$ cat test_noindent.mysql
mysql-> select * from firstLine;
mysql-> select * from secondLine;
mysql-> \help contents
mysql-> \h contents

# see outputs in screenshot below
$ pygmentize -l psql -F gobble:n=4 test.mysql

$ pygmentize -l psql test_noindent.mysql

[muzimuzhi]

minted option gobble=4 is passed to pygments executable pygmentize as CLI flag -F gobble:n='4'. But unfortunately, pytments "filters are applied after lexing" (from https://github.com/pygments/pygments/issues/585#issuecomment-526848356).

Also see pygments doc on filters

An arbitrary number of filters can be applied to token streams coming from lexers to improve or annotate the output.

Hence if an indented PostgreSQL console (psql) should be correctly tokenized, it's PostgresConsoleLexer to blame.

[goyalyashpal]

hi! thanks a lot for such thorough investigation..

if an indented PostgreSQL console (psql) should be correctly tokenized, it's PostgresConsoleLexer to blame.

ehm, no. 'cz the console input-output is never indented i.e. false at the "if" itself. so, the pgconsole lexer is perfectly fine the way it works.

minted option gobble=4 is passed to pygments executable pygmentize as [a filter option]

if minted can autogobble, then can it not also gobble?

But unfortunately, pytments "filters are applied after lexing" (from pygments/pygments#585 (comment)).

the issue lies here only.

[goyalyashpal]

regardless of where the solution is supposed to come; i think this behaviour is worth a place in documentation:

• "gobble as a filter to pygments which's then applied there after lexing" , or more clearly & succinctly as:
• "autogobble: a minted's preprocessing option, gobble: a pygments' post-lexing filter"

[gpoore]

gobble and autogobble behave differently because only gobble is a Pygments filter. All of this will change in minted v3.0, since there will be a minted executable where filter-type behavior can be implemented separately from Pygments, both before and after lexing. At that point, all gobbling can be implemented in a uniform fashion. I'm about to release a new LaTeX data serialization package that will be the foundation of much of minted v3.0, and then will be able to start working on the new executable.