Convert Docs from recommonmark to MyST-Parser

[cooperlees]

Read the docs is moving to MyST-Parser. So I guess we need to. (Haven't done homework yet)

readthedocs issue: https://github.com/readthedocs/recommonmark/issues/221

cc: @ichard26

[felix-hilden]

As an aside, if I may inquire: why markdown in the first place? Is it simply for the (debatably) better plain text readability or some historical reason?

[cooperlees]

I prefer the markdown syntax and it’s simplicity. I didn’t make the choice here, but do for other projects I maintain. I’m pretty sure black just using Markdown was due to GitHub’s MD love.

Why the format question? Do you feel there is some advantage converting docs to another format? If so, what?

[felix-hilden]

Out of genuine curiosity! I've heard arguments both ways, but for some reason I've thought that RST is preferred in the Python community - a claim that I realised has no real basis 😅 I'd imagine that the banger argument for RST is roles which Sphinx of course extends, but I'm guessing any MD->RST tools also provide something for that.

Most likely nothing so valuable that would warrant converting the whole documentation over. But aside from not recognising Sphinx roles I've not had any issues in, say, having a readme as RST. Hmm, let me figure out MyST a bit before causing any more chaos to this very well-defined, technical issue 😄 Thank you for the reply though!

[hukkin]

@cooperlees if you need a MyST compatible replacement for Prettier, then AFAIK mdformat is the only option.

A .pre-commit-config.yaml configuration more or less equivalent to the current Prettier config would look like:

- repo: https://github.com/executablebooks/mdformat
  rev: 0.7.4
  hooks:
    - id: mdformat
      args: [--number, --wrap=88]
      additional_dependencies:
        - mdformat-gfm
        - mdformat-myst

disclaimer: I'm the author

[ichard26]

FYI, @cooperlees isn't best person to ping for this... as a *the* documentation person I am usually the best option. Cooper opened this issue since they were the one to notice this deprecation first.

Also, thanks for the suggestion, I definitely want to use MyST syntax and I forgot that Prettier obviously wouldn't support it, so I'll look into it (one day).

[ichard26]

While there's a significant chance I'll end up being the BDDFL (Benevolent. Documentation. Dictator. For. Life) for this project and end up pushing many changes without consultation or feedback (to be clear I don't want that!), let's see if I can get some feedback.

So with this deprecation there's three options worth considering:

1. Migrate all documentation to MyST, including the restructuredText documentation (this option implicitly requires usage of MyST's specific syntax extensions)
2. Migrate all Markdown documentation to MyST and start using MyST specific syntax extensions for the CommonMark standard (also drop the prettier autoformatter for mdformat)
3. Migrate all Markdown documentation to MyST-Parser and DON'T use any MyST specific syntax extensions

	Pros	Cons
Option 1	Only requires one formatter to handle docs (mdformat) Simplifies contributing to documentation since Markdown is easier to write than RST	Requires an one-time cost of porting over the RST docs to MyST
Option 2	Doesn't need as much work as Option 1, yet our Markdown docs can be as expressive as our RST docs	We end up with this somewhat mix of RST and MyST, complicating changes to the docs. Also RST is hard to work with, especially for less experienced contributors (which I argue is also an important contributor base to help)
Option 3	We can stick to our usage of only prettier for our docs since "standard" Markdown can be formatted by it	Other than using a new shiny and better supported Markdown parser, our Markdown docs will continue to be unable to express many constructs supported by RST.

terminology-note:

• Markdown is a lightweight markup language for creating formatted text, but due to lack of a clear standard, has involved into a large family of similar (yet usually incompatible) languages.
• CommonMark is a rationalized version of Markdown syntax that aims to standardize Markdown
• MyST is a version of Markdown that extends off of the CommonMark spec
• MyST-Parser is a general purpose Markdown parser, notable for implementing MyST

I personally prefer option 1 since I like MyST over RST and I don't mind paying the up-front cost of porting our docs. Option 2 is a middle of the road approach, and it's fine, but the mix is annoying without much benefit. And for option 3, I mean, it's pretty much the "do nothing" approach which is nice and all, but like, if we have to switch to MyST-Parser anyway, let's use its handy benefits while at it.

Thoughts?

[hukkin]

I also prefer "Option 1". If there's a way to avoid documenting in two different markup languages it should be the way to go IMO.

Regarding

We do end up needing two formatters, one for RST, and another for MyST (while prettier supports Markdown, it doesn't support MyST's syntax extensions) (i.e. both prettier and mdformat)

Hmm, are you sure that prettier supports RST? Or maybe you don't even imply that and I misunderstand? At least Prettier docs don't mention RST support anywhere.

[hukkin]

Btw https://github.com/executablebooks/rst-to-myst exists for RST -> MyST conversion, which might be helpful. It's still in development though and according to its author "just about ready"

[ichard26]

Hmm, are you sure that prettier supports RST?

Nope, I was wrong :-). Thanks for correcting me!

[ichard26]

@hukkinj1 question about the MyST support of mdformat, does it support the optional colon_fence extension? I worked around it by using the standard backtick code fence syntax for directives but I'd liike to use colon fences some time in the future. Thanks in advance!

[hukkin]

The support is currently only for syntax that is enabled by default in myst-parser, so unfortunately no colon_fence yet.

Edit: All currently supported syntax extensions are listed here https://github.com/hukkinj1/mdformat-myst/#description

[ichard26]

@hukkinj1 thanks for the clarification. If my desire grows enough I might consider adding support myself if you'd welcome it :)

[hukkin]

I might consider adding support myself if you'd welcome it

:+1: PRs are always welcome :smile: Note that I think it's good that there's a way to enable only the default extensions, so support for this would likely mean another plugin. Maybe better to plan and discuss this in another issue (and issue tracker) first.

[ichard26]

@hukkinj1 another question (I'm sorry but I'm been actively using mdformat and been hitting some road blocks), are [//]: # "content" comments supposed to be stripped? I took a look at the mdformat style docs but couldn't find anything about comment handling. Thank you once again!

[hukkin]

I'm sorry but...

No problem, I'm glad to be able to help.

[//]: # "content" is not a comment, it's actually a link reference definition. Mdformat removes link reference definitions if there is no reference link referencing them.

For a comment, MyST provides the % this is a comment syntax. An alternative (pure CommonMark compatible) solution is an HTML comment <!-- this is an HTML comment -->.

[ichard26]

@hukkinj1

Cool, I'm learning something new everyday! Yeah, I just used the myst alternative since github readability is already thrown out the window :)

I should probably read the CommonMark standard.

[JelleZijlstra]

@ichard26 can this be closed now?