Switch to CommonMark

[rwols]

When rendering a list, mdpopups seems to prefer to have the list prepended with two newlines:

Python 3.3>>> mdpopups.md2html(view, 'foo:\n\n- hello\n-there\n')
'<p>foo:</p>\n<ul>\n<li>hello<br />\n-there</li>\n</ul>'

Indeed a single newline doesn't render a list:

Python 3.3>>> mdpopups.md2html(view, 'foo:\n- hello\n-there\n')
'<p>foo:<br />\n- hello<br />\n-there</p>'

There is however this markdown syntax where you can put two whitespace characters at the end of a line, and supposedly this should result in a linebreak:

Python 3.3>>> mdpopups.md2html(view, 'foo:  \n- hello\n-there\n')
'<p>foo:<br />\n- hello<br />\n-there</p>'

Is this also supposed to render a list? Note that the clangd language server seems to utilize this for rendering its lists.

[facelessuser]

The two spaces would create a linebreak in a paragraph, the next line be considered a continuation of the paragraph. Not all parsers handle it the same, but there are a lot (especially Markdown.pl -- which is the reference fro Python Markdown):

See Markdown comparisons: https://johnmacfarlane.net/babelmark2/?text=foo%3A++%0A-+list%0A-+list

[rwols]

I'll try communicating with the clangd maintainers, but this may turn up more often seeing as vscode interprets it as a list.

[facelessuser]

Yeah, commonmark handles this, but commonmark parses the Markdown in a very different way. There may be a Python Markdown plugin that handles this...I'd have to check.

[rwols]

I'm upgrading this issue to "please drop Daring Fireball markdown", because:

1) Github-flavoured markdown is used on GitHub. It's therefore the default choice for a vast amount of technical people. It is based on CommonMark. 2) VSCode uses commonmark.js, another large player that therefore promotes CommonMark. 3) The Daring Fireball spec is vague and incomplete in many places, the CommonMark spec is precise and unambiguous.

Some use-cases for direct CommonMark parsing:

1) We can fetch github readme's from the internet and put them into mdpopups.new_html_sheet. 2) Language servers are eventually going to depend on more niche CommonMark properties (essentially, they're going to be relying on the commonmark.js implementation in VSCode).

[rwols]

There's also py-gfm which could work https://pythonhosted.org/py-gfm/

[facelessuser]

There's also py-gfm which could work pythonhosted.org/py-gfm

I'm pretty sure you are only going to get a commonmark-ish, github-ish feature set from that one. It even states:

This implementation does not support all of GFM features.

You're probably going to need something more like https://pypi.org/project/commonmark/ or maybe a different one. Then you'll have to build GFM features on top of that (if you want full GFM which is not standard even to CommonMark). So, likely, maybe a pure CommonMark with fenced code built on top (unless CommonMark has defined those as well).

Ugh, maybe we need to implement auto-linking too as I don't think that is standard in CommonMark. So basically, I need to explore CommonMark options and find out which one is easiest enough to customize.

Anyways, this is a lot of work, not impossible, but it'll be work. If we want a commonmark library to be generally available in Sublime (and not vendor it here) I'll request that someone else maintain and provide that dependency to Sublime. I'll have to decide if I'm going to vendor or not once I can figure out what library to use.

When I have some time, I'll look into this more. I have to finish up my work in the Sublime ColorHelper plugin first. And I'll probably need a break after that too. So when I get some time, I'll revisit this and see what can be done.

Anyone who wants to start evaluating commonmark libraries for Python can provide details here. It may help me get moving a big faster on this.

[rwols]

Thanks for considering it.

Anyone who wants to start evaluating commonmark libraries for Python can provide details here.

What kind of evaluation reports do you find helpful? I'm certainly willing to put some effort into this.

[facelessuser]

Honestly, I'm looking to see what kind of options are available for Python. I vaguely remember there being at least two CommonMark libraries for Python (one I think started with an M or was it an L 🤷).

I'd like to understand how flexible they are in terms of customization as we have to probably add Tables (as Sublime doesn't support rendering these, maybe this isn't really needed right now), fenced code (with highlighting), and auto-linking at the very least. I'm pretty sure that none of these are found in the commonmark spec. Maybe there are some of these customizations already available via 3rd party extensions, plugins, wrappers, etc.

That's probably a good starting point. Afterwards, I'll need to figure out the best way to integrate them. To keep the API basically the same, we may be able to simply create a Python Markdown extension that just parses everything in CommonMark or we just sidestep Python Markdown all together.

[facelessuser]

Mistune was the other library I was thinking about: https://github.com/lepture/mistune.

I've tried none of the CommonMark parsers, but evaluating this and https://pypi.org/project/commonmark/ and at least determining which of these will work best for us will probably be the first task.

[facelessuser]

I'm posting this as a possibility: https://github.com/executablebooks/markdown-it-py

I have no idea if it works in py3.3, but it claims commonmark compliance, seems maintained, has been extended or can be extended to do basic things like fenced code, auto linking URLs, etc.

No doubt there are are probably some incompatibilities between some features in the current parser and this one, but it can potentially be used as an alternative.

Now, if I have to do some extensive backporting to 3.3, this may wait until py3.8.

I also don't know how easy it is to inject the code highlighting, but hopefully it isn't too bad.

[facelessuser]

Let's be honest, this may wait till py38 anyways :).

[facelessuser]

Looks like code support is probably done from extending the renderer: https://markdown-it-py.readthedocs.io/en/latest/architecture.html#renderer.

I imagine we can trigger off the token's type='fence' and tag="code" properties, run it through our existing syntax highlighter and return the results.

I imagine we will just enable the reasonable Markdown parsing options and just allow people to enable commonmark parsing. Beyond picking Pygments vs Sublime Code highlighting, I don't imagine we will expose much. TBH, a lot of the "extra" stuff is not CommonMark, but certain things I'm considering a must-have regardless (namely fenced code and auto-linking of URLs). Table support is probably nice to have even though Sublime can't render tables...yet.

The idea of providing CommonMark right now is to give a standard-compliant option, so it won't be too configurable. This will save me from developing an interface for configuring it as I think I'll have to provide a decent amount of code to make it as easy to configure as Python Markdown. Maybe in the future, we can ween ourselves off Python Markdown, but I think there are still things that are easier to do with Python Markdown (like parsing Markdown in HTML block tags) than they are to do in this parser, but I imagine we can one day work around those things.

I'm optimistic looking over the docs.