Dollar signs in Markdown code blocks perturb rendering

[slel]

In a CoCalc markdown document, type a code block spanning at least two lines

Change

    $ cd
    $ cd
    $ cd

and see it rendered as if the second line had 4 extra spaces at start.

Change

    $ cd
        $ cd
    $ cd

Another way to phrase it is that the second line gets indented for free, so that with the second line unindented

Change

    $ cd
$ cd
    $ cd

the code block pretends it was indented.

Change

    $ cd
    $ cd
    $ cd

See screenshot:

[slel]

This happens only when dollar signs are involved. Maybe related the recent commit to fix Markdown and LaTeX: https://github.com/sagemathinc/cocalc/commit/5aeb5481fd14011cd77438ef84f834a1bee5d449

[williamstein]

This is one of those edge cases that is extremely difficult to parse in a way that doesn't break other things. It is also broken in precisely the same way in vanilla upstream Jupyter:

I recommend using ``` rather than indentation for code. It's far more robust. Markdown isn't the best spec.

It may make sense to close this as "cantfix" for now, since we do aim for jupyter compatibility as much as possible, and it may be the case that not all sensible math can be parsed in markdown.

[slel]

Thanks for the workaround of using triple-backquotes, somehow I did not have that in mind.

[slel]

Follow-up: say one wants a code block inside a numbered list.

One can achieve this either using triple-backquotes:

1. In a terminal, run

$ cd
$ ls
$ ls -halF
$ cd
```

2. What did you observe?

or extra indentation:

1. In a terminal, run

        $ cd
        $ ls
        $ ls -halF
        $ cd

2. What did you observe?

In a GitHub issue they get rendered as one would expect.

The one with backquotes gives:

1. In a terminal, run

   $ cd
   $ ls
   $ ls -halF
   $ cd

2. What did you observe?

The one with extra indentation gives:

1. In a terminal, run

   $ cd
   $ ls
   $ ls -halF
   $ cd

2. What did you observe?

But in CoCalc the rendering is messed up by the dollar signs.

The one with dollar signs looks like:

1. In a terminal, run

   $ cd
       $ ls
   $ ls -halF
       $ cd

2. What did you observe?

The one with extra indentation looks like:

1. In a terminal, run

   $ cd
           $ ls
   $ ls -halF
           $ cd

2. What did you observe?

[slel]

Could an existing markdown renderer be used that gets this right?

• the one at GitLab CE?
• the one at CodiMD?
• the one at Gitter?

[mforbes]

I think issue 2667 is the relevant issue in the Jupyter project.

[slel]

See also this JupyterLab pull request:

• jupyterlab/jupyterlab#10867

[williamstein]

Last week I spent a long time properly fixing all of this by fully re-implementing our Markdown rendering using a custom rendering pipeline and parser. It's what's used now for the new share server, and also internally for WYSWIG Slate, and I can switch it on for most of cocalc, but have NOT done so yet, since there are likely still some subtle edge cases to fix.

[williamstein]

I checked, and the new code I wrote (and also the WYSIWYG editor) both still have this problem. It's very likely literally caused by code from Jupyter classic, which I copied into CoCalc specifically for this parsing.

[slel]

Apparently solved by the latest rewrite of markdown rendering in CoCalc.