backtick in a doc breaks the parser

[stew]

if you have a backtick in a doc, the parser fails and blows past the fold to place the error somewhere seemingly randomly after the fold looking for the matching backtick

x = {{ ` }}

---

> "hello"

[sellout]

scratch/main> builtins.mergeio

I have an in-progress change that fixes this and I’m assuming the desired behavior should match CommonMark (since we seem to be leaning that way for other questions, like #5255).

In this case, that means

x = {{ `putText stdOut }}

---
> "hello"

would parse as Word "`putText stdOut".

For a slightly more complicated question … Verbatim (backticked) text can currently “escape” the closing Doc delimiter (and other syntax).

x = {{ To close a Doc, use `}}` }}

scratch/main> add

  ⍟ I've added these definitions:

    x : Doc2

scratch/main> display x

  To close a Doc, use `}}`

I don’t think we have any other mechanism that can do that currently.

Other inline syntax doesn’t overrun }}

y = {{ To close a Doc, use * }}

scratch/main> add

  ⍟ I've added these definitions:

    y : Doc2

scratch/main> display y

  To close a Doc, use *

But it also doesn’t allow us to capture }}.

y = {{ To close a Doc, use *}}* }}

  Loading changes detected in scratch.u.

  I got confused here:

      1 | y = {{ To close a Doc, use *}}* }}

  I was surprised to find a }}* }}
   here.

And CommonMark escape syntax is handled literally.

z = {{ To close a Doc, use \}\} }}

scratch/main> add

  ⍟ I've added these definitions:

    z : Doc2

scratch/main> display z

  To close a Doc, use \}\}

So what should we do for that case? I’m thinking support for CommonMark backslash escapes is good, but (in CommonMark) those don’t work in verbatim blocks … so how do we get a monospace }} into a Unison Doc? The current workaround is

q = {{ To close a Doc, use {{ docCode (docWord "}}") }} }}

scratch/main> add

  ⍟ I've added these definitions:

    q : Doc2

scratch/main> display q

  To close a Doc, use `}}`

This doesn’t complain about the }} nested in the transclude, because anything that embeds Unison code calls the Unison parser, passing it the expecting closing tokens (}} in this case) … and the Unison parser only checks for those at a valid expression boundary.

But that also means that Unison code can swallow the fold …

r = docWord "whoops, forgot to close this

---
nothing below here should be evaluated"

scratch/main> add

  ⍟ I've added these definitions:

    r : Doc2

scratch/main> display r

  whoops, forgot to close this

  ---
  nothing below here should be evaluated

Which is a similar bug on the other side of the coin.

(BTW, this transcript uses the idempotent syntax from #5406, which makes transcript outputs runnable)