Allowing markup with code examples? Alas, no.

[lizmat]

I've come to the conclusion that it is impossible to generically allow markup inside code blocks that have syntax highlighting.

I'll try to explain why:

1. Syntax highlighting depends on being able to compile a given piece of source code.
2. Code with markup inside of it is not valid source code, so the markup needs to be removed before the code ca be compiled and syntax highlighted.
3. Removing the markup makes it impossible to trace back where the additional handling for the markup needs to take place, except for the original content of the markup.

My current implementation depends on being able to find the original content. However, since the markup has no relation with the AST, and thus with the syntax highlighting, it is very likely it cannot be found, so the markup handling / substitution can not be applied.

When I originally implemented this, I tested without active highlighting, and without recursive markup. An example:

mB<y> $a V<=> 42;

would be perfectly legal markup. The markup would be removed, the code compiled and syntax highlighted`, and we'd get with HTML highlighting:

<span style="color:magenta;">my</span> <span style="color:cyan;">$a</span> <span style="color:yellow;">=</span> <span style="color:red;">42</span>;

Now, for the V<> handling, it would look for the first "=" in the result (as the markup was around the first = in the source), and then does the substitution { "<V>" ~ .atoms ~ "</V>" }

<span style<V>=</V>"color:magenta;">my</span> <span style="color:cyan;">$a</span> <span style="color:yellow;">=</span> <span style="color:red;">42</span>;

instead of:

<span style="color:magenta;">my</span> <span style="color:cyan;">$a</span> <span style="color:yellow;"><V>=</V></span> <span style="color:red;">42</span>;

It gets worse with the mB<y> case. And even worse still with recursive markup.

I've now spent about a week on this, and I don't see a solution. And restricting markup to entities matching the AST, is not a solution either, as you can't really tell from the code how the AST will be built, or can we guarantee that that will always stay the same.

So I think we either need to disallow markup in code blocks altogether, or that specification of :allow<B> implies :!syntax-highlighting. In which case source code should be treated as documentation (which basically means a no-op, as any renderer will have already got the AST for that).

[lizmat]

@coke @finanalyst @thoughtstream

Comments? Am I missing something that would allow markup and syntax highlighting?

[finanalyst]

@thoughtstream, @lizmat has unfortunately come to the conclusion I arrived at quite a long time ago - that :allow and highlighting are generally incompatible. I was sincerely hoping to be found wrong, and Liz has found a way to get some partial resolution. But not a generic solution.

I have found it very difficult to handle :allow type options when an external highlighter is used for other languages, such as Haskell or Python.

However, the specification does permit the renderer to choose how to resolve a conflict when both :allow and :lang are given.

So, I think I shall adopt @lizmat's suggestion for Rakuast::RakuDoc::Render, namely that :allow implies :!syntax-highighting.

@coke, FWIW, there are only four sources in docs that contain :allow apart from the two (three) documents that specify :allow. @cfa (I think) removed :allow markup from most of the other sources about a year and a half aog.

Here is the output from rak in my local docs, which contains a symlink to rakudoc.rakudoc source:

$ rak ':allow' . --files-with-matches
Language/pod.rakudoc
Language/operators.rakudoc
Language/rakudoc.rakudoc
Language/quoting.rakudoc
Language/create-cli.rakudoc
Language/modules.rakudoc
Type/RakuAST/Doc/Block.rakudoc

[thoughtstream]

I agree that it seems impossible to annotate the AST of a =code block with mark-up. Because the structure of a mark-up block need not align with the syntactic structure of the contained code (or even with its lexical structure).

In my view, being able to deliberately and explicitly highlight the important parts of a code block with B<> or I<> is extremely important, whereas being able to have code auto-syntax-coloured is merely "nice to have" (for some people).

Being able to explicitly highlight specific parts of a code block enables an author to focus their readers' attention on the key point of the code sample. So :allow is a vital tool for effective communication in technical writing.

But having explicitly highlighted code also be syntax-coloured actually undermines the goal of focusing the readers' attention, because syntax-colouring injects a vast amount of distracting visual noise into the code (which, of course, is my personal opinion of all syntax-colouring...even when the code isn't being deliberately highlighted via mark-up! ;-)

Hence, I have no problem with us specifying that a renderer may treat :allow and :syntax-highlightingas being mutually exclusive. And that a renderer may assume that :allow implies :!syntax-highlighting.

However, I don't think it's necessary to specify that :allow and :syntax-highlighting must always be treated as mutually exclusive. Or that a renderer must ensure that :allow implies :!syntax-highlighting.

After all, someone might eventually find a way to accommodate simultaneuous explicit markup and automatic syntax-colouring. And there's no need for us to forbid them from providing both, if they can.

Damian

[finanalyst]

@thoughtstream @lizmat It seems to me that we are all in agreement here in principle. Would you please look at the current document to ensure that the text does in fact clearly convey this agreement?

[thoughtstream]

Sorry @finanalyst, but I find I'm confused. Did you want us to look at the pull request, or the current document, or some other update that I can't seem to find?

(Apologies, but it's quite late here on the dark side of the planet and I'm apparently also suboptimally bright right now. I'll try again in the Antipodean morning :-)

[finanalyst]

@thoughtstream A bit of a mix-up on my end. I saw that Liz had approved the PR, and for some reason thought it had been merged. So I did not differentiate between the PR and the current document. My mistake.

I have just reviewed the new text including the PR and I think that it reflects this thread correctly. So I am merging it.

Closing this issue as resolved

[thoughtstream]

No worries, @finanalyst. I'm just happy the issue is resolved.