Re-connecting from #55. I apologize for that one, I didn't read the README... I think I got it right this time!
This PR will update the tokenization of source code listings using general-purpose packages, such as listings.
In particular, it will improve the grammars for detecting the particular language of the listing.
The grammars were defined so that the only way it would detect the language is when it was provided (lower-case only) as the second argument in braces, e.g:
\begin{lstlisting}{python}
However, this does not conform with the actual usage of these packages. In the case of listings, for example, the correct syntax is
\begin{lstlisting}[language=Python]
Additionally, in cases where the language was not specified, all code inside the listing would be tokenized and syntax-highlighted as LaTeX text.
Contribution
In b2bc75ae740cb5f60410c77c9998385ed775580f, build\generate-grammar-blocks.js is updated to:
replace the standard argument tokenization with custom rules that check for the language;
add case-insensitivity to language list;
and add Matlab/Octave to the list of languages.
Further, syntaxes\data\LaTeX.tmLanguage.json is updated to:
make the "default" language style markup.raw.verbatim.latex, just like \verbetc;
add support for the pygmentedenv. from the packages pygmentex and texments.
An example of the resulting regex-string for the new rule can be seen in Debuggex here.
Additionally, 697409d850f8d0945aed28e56a4a212d826ee4f6 adds support for a few more one-line commands for including external files. The path name is tokenized as a string. This covers the same ground as #52
Finally, 1803dfa91657329f064236517323a9142254645a adds a few lines to the README.
Known issues
This does not handle the case of using "dialects" inside listings, i.e. [language={[ansi]C}, ...]. Currently, this causes it to be treated as verbatim.
This does not handle the case of breaking up the optional arguments over several lines. Even though \sshould capture newlines...
This does not fix the issue where missing delimiters cause the syntax highlighting to escape the \end{...} line. But this is not a new issue...
nilsjor
commented
1 year ago 
Re-connecting from #55. I apologize for that one, I didn't read the README... I think I got it right this time!
This PR will update the tokenization of source code listings using general-purpose packages, such as
listings. In particular, it will improve the grammars for detecting the particular language of the listing.Previously, three environments were supported:
lstlisting(from listings),minted(from minted),pyglist(from verbments).The grammars were defined so that the only way it would detect the language is when it was provided (lower-case only) as the second argument in braces, e.g:
However, this does not conform with the actual usage of these packages. In the case of
listings, for example, the correct syntax isAdditionally, in cases where the language was not specified, all code inside the listing would be tokenized and syntax-highlighted as LaTeX text.
Contribution
In b2bc75ae740cb5f60410c77c9998385ed775580f,
build\generate-grammar-blocks.jsis updated to:Further,
syntaxes\data\LaTeX.tmLanguage.jsonis updated to:markup.raw.verbatim.latex, just like\verbetc;pygmentedenv. from the packages pygmentex and texments.An example of the resulting regex-string for the new rule can be seen in Debuggex here.
Additionally, 697409d850f8d0945aed28e56a4a212d826ee4f6 adds support for a few more one-line commands for including external files. The path name is tokenized as a string. This covers the same ground as #52
Finally, 1803dfa91657329f064236517323a9142254645a adds a few lines to the README.
Known issues
listings, i.e.[language={[ansi]C}, ...]. Currently, this causes it to be treated as verbatim.\sshould capture newlines...\end{...}line. But this is not a new issue...