\include within LaTeX environment scrambles conversion

[uliska]

Pandoc 2.0.6, installed from the .deb package

I am using Pandoc for a Markdown->LaTeX->PDF chain and use custom LaTeX in the Markdown file. I have a case where the content of a LaTeX environment scrambles the generated LaTeX code:

\begin{lilypond}[nofragment]
\include "oll-core/package.ily"
\loadModule stylesheets.span
{
  cis'4 \span blurred { d'8 d' } eis'4
}
\end{lilypond}

(making use of functionality from the lyluatex package, the environment content is passed to LilyPond, an external program).

This will be rendered to the following .tex:

\textbackslash{}begin\{lilypond\}{[}nofragment{]}
\include ``oll-core/package.ily'' \loadModule stylesheets.span \{ cis'4
\span blurred \{ d'8 d' \} eis'4 \} \textbackslash{}end\{lilypond\}

where nearly everything is escaped, which is of course wrong and makes the .tex code fail.

---

It can be made to work by one out of:

• commenting the \include line
• insert at least two tokens before the \include

\begin{lilypond}[nofragment]
\noop something
\include "oll-core/package.ily"
\loadModule stylesheets.span
{
  cis'4 \span blurred { d'8 d' } eis'4
}
\end{lilypond}

However, it doesn't work with a simple \noop (regardless of the fact that this is a fake command anyway)

---

Shouldn't Pandoc, when it encounters a \begin{something} pipe its content through regardless of what it contains?

[mb21]

Can you try again with the latest pandoc release?

the environment content is passed to LilyPond, an external program

btw. this sounds like a usecase for a pandoc filter http://pandoc.org/filters.html

[uliska]

Can you try again with the latest pandoc release?

This didn't change anything.

I have found a workaround by inserting noop = {} which is a consequence-less assignment in LilyPond. But it still is a hack.

the environment content is passed to LilyPond, an external program

btw. this sounds like a usecase for a pandoc filter http://pandoc.org/filters.html

I know, and there even is a filter for LilyPond that makes use of lyluatex (https://github.com/jgm/pandocfilters/blob/master/examples/lilypond.py). But that uses a totally outdated version of the package, which we pushed from something like 0.1 to 1.0b since January.

[mb21]

But that uses a totally outdated version of the package

feel free to update it and make a pull request :-) (btw, doesn't it call out to whatever version of lilypond is on your python path?)

Until this particular parsing latex in markdown bug is fixed, you can also use generic raw attributes to embed complex LaTeX in Markdown, like ```{=latex}...

[uliska]

But that uses a totally outdated version of the package

But that uses a totally outdated version of the package

feel free to update it and make a pull request :-)

Well, I've donated around 80 hours of unpaid work to lyluatex

and I can't afford to continue on that track before having spent some more paid hours on other projects ;-)

(btw, doesn't it call out to whatever version of lilypond is on your python path?)

That depends on the configuration, but I was talking about the lyluatex package. The filter generates .tex code to make use of lyluatex, and that will probably not work too well since lyluatex has so fundamentally been changed in the meantime.

Until this particular parsing latex in markdown bug is fixed, you can also use generic raw attributes to embed complex LaTeX in Markdown, like {=latex}...

Oh, thank you, I wasn't aware of that, and it seems to work for my case. It's still a workaround, but a much cleaner one that having to insert arbitrary dummy code into the environment.

---

So for my current use case this can be considered solved, but I would still consider it a parsing bug.

[jgm]

I'm assuming that the lilypond environment redefines \includeso it can take a double-quoted argument, without braces? That doesn't work in latex in general, and the latex parser is stumbling on this.

[uliska]

No, but your comment probably gets me to the right point: lyluatex doesn't do any parsing of the environment's content, but it goes to some length protecting it so it can be piped as a pure string into a LilyPond document. LilyPond itself also has \include, but of course with a totaly different syntax.

Does Pandoc parse the content of a \LaTeX environment if it encounters one? I thought that it would (should) simply pass that content along to LaTeX. But given the pointer to the "generic raw attributes" I have the impression that Pandoc continues parsing the environment content.

If that is the case then I withdraw the claim this is a parsing bug. Instead it would be a caveat to document for lyluatex users who want to include scores when using Pandoc.

[jgm]

Yes, by default, pandoc parses the contents of unknown environments. This is overridden for certain environments known to be "verbatim." We could add lilypond to that list, of course.

[uliska]

I'm not sure. There may be various names that might be involved. Maybe it's better to pull that back into lyluatex's documentation.

Do you have an opinion @jperon ?

[jperon]

I think that lilypond (and ly) really should be treated as a verbatim environment, as I don't see any reason to parse it and treat it as markdown or latex.