Markdown+MyST, and, or, rST?

[consideRatio]

I think there is a case to be made for using Markdown systematically across our repos, so I'll try make it.

• Markdown is well known. Jupyter, GitHub, and Discourse all use Markdown, while rST is far less known in the space we work I feel. We will get more contributions and help with maintenance if our documentation is technically comprehensive to the users.
• Markdown + MyST are technically sophisticated. In the past, I recall the choice of rST related to limitations in Markdown, but I believe those are overcome with the addition of MyST parser.
• Less is more. By choosing either rST or Markdown, the JupyterHub community need to learn less technical parts, and if we choose to use Markdown, what they learn will have synergy with Jupyter / GitHub / Discourse.

I raise this issue because I'm quite excited by contributing to yuvipanda/jupyterhub-ssh and want to do some maintenance on the jupyterhub/binderhub repo, but both adopt rST and I'm just not as familiar with rST as with Markdown, and it creates some mental friction in my mind. So that led me to raise this question - what are your thoughts about going all in on Markdown + MyST as tooling for documentation?

Ping @jupyterhub/jupyterhubteam!

[willingc]

I think that it doesn't have to be an all or nothing decision. Whether you use markdown or rST, with the right plugins, source can be mixed and matched. I would personally opt for using either even in the same project since docs written are better than unwritten.

@consideRatio If you are writing docs from scratch, writing them in Markdown and then convert using pandoc to rST works pretty well. If you are editing a rST doc, get a good plugin for VSCode (or favorite editor) that lints and /or suggests syntax and you will be productive. Most docs that I write these days is in Markdown but there are some cases when I am glad that rST is available when I need it.

[consideRatio]

@willingc :heart:! I'm happy you are chiming in!

there are some cases when I am glad that rST is available when I need it.

Are there typical situations when rST is better for you? I recall that rST has been better to reference other documents for example, but that is the only example I have clear in mind.

get a good plugin for VSCode

Are you also using VSCode and happen to have a recommendation? =) There are soooo many plugins and I could not find any obvious choice.

docs written are better than unwritten.

I feel this way as well. You have known me since i entered the open source world, and since then I have not overcome my rST friction over ~3 years of exposure to it. I think it is because I use it too little but still need to use it here and there, and I never really get proficient and wonder if other users spending less time in the open source domain are slightly held back when they are exposed to rST compared to Markdown.

Its tricky to evaluate, so I'm mostly speculating though.

---

I hope you are well @willingc, are you in the San Diego area as GitHub indicates? Hi from Stockholm!

[betatim]

I have a strong preference for us to not spend time converting documentation from one format to another or adding to the infrastructure used for building our documentation. I think recuperating the time spent on this kind of infrastructure work by being more efficient in adding docs is very hard (if we spend 2h on transforming docs, then after 2h of effort we have spent 2hrs but the docs are still no better than before. To make up these 2hours we need massive productivity boost and a lot of hours spent doc'ing).

I think rst is a good choice as it is more maintenance friendly when it comes to cross referencing material which I think is a important feature of technical documentation (build a "web of topics" for users). In Markdown people end up hardcoding URLs when cross referencing which causes broken links/tedious work later. I don't think myst is more widely known in the community as a whole. I use rSt and myst rarely enough that I have to look up the more advanced directives in either (for example embedding an image in the page is something I need to google each time).

I think editing an existing rSt document is Ok as you can usually find examples of what you need in it. For authoring one from scratch I will start adopting Carol's trick of MD -> rSt via pandoc.

[choldgraf]

Hey all - I am obviously biased here, but will try to give some points that are relevant:

To make up these 2hours we need massive productivity boost and a lot of hours spent doc'ing).

To me there are three potential groups to think about here:

• Team members who are comfortable with rST
• Team members who are not comfortable with rST but are comfortable with markdown
• Non-team members on GitHub (who I think we can assume know markdown, but not rST)

To me, it is worth using markdown because people in group 1 are almost always also familiar with markdown, and thus it makes the documentation source more accessible and editable to a larger group. This is particularly true for outside committers, as in my experience the vast (vast, vast) majority of users don't know rST and it is intimidating to them to contribute to docs because of it.

I think rst is a good choice as it is more maintenance friendly when it comes to cross referencing material which I think is a important feature of technical documentation

All of this is possible in MyST Markdown. MyST and rST are nearly at feature parity with one another in Sphinx (the only non-overlapping syntax features are ones I've never used in rST anyway). For example, to cross-ref a doc, you could write {doc}path/to/doc.md` (aka, a Sphinx role), or you could write(which MyST will interpret as a Sphinx:any:` role).

I agree with your general point that the features of a true "documentation engine" like Sphinx are useful enough to keep using rST if it were not possible in markdown, but it is now possible in markdown so I think it's worth re-considering.

for example embedding an image in the page is something I need to google each time

IMO that's a good example where most people wouldn't need to look something up if we used markdown. I'd bet that you know ![]() image syntax by heart, as many do. The same is true for a lot of syntax in rST vs. markdown (for me, anyway)...in rST I have to look things up, in markdown I "just know it".

To make up these 2hours we need massive productivity boost and a lot of hours spent doc'ing).

It's worth pointing out this "in development" tool: https://github.com/executablebooks/sphinx-tomyst

which will auto-convert rST into MyST markdown in batches. Perhaps that can be piloted on a few smaller jupyterhub repositories to see how it goes. I'd also note that to me the main goal of markdown is to make documentation source more accessible to those outside of the repository, who are probably not familiar with rST. It's hard to quantify but I wonder if we'd make up those two hours in the form of other users contributing documentation more often. I can say anecdotally that many users have contributed docs to JupyterBook, and most of them don't know rST and likely may not have made the contribution if we required them to write it.

Anyway, in summary - I personally much prefer markdown to rST - I think it's more accessible for a broader community and also just much easier for me to remember how to write. Because of MyST, there is no feature in rST that I wish to use that is not possible in markdown, so for my personal use, it is a straightforward choice.

For the team, my personal approach has been: "if I'm writing docs from scratch, write markdown, if I'm making heavy edits to an rST page, convert it to markdown, if I'm making small edits to an rST page, leave it as rST"

[consideRatio]

Thank you for the technical comparison @choldgraf!

My motivation of discussing this stems mainly from I believe it will help our lives as maintainers easier indirectly, because I think we can help others help us maintain our documentation better by using Markdown+MyST compared to using rST, which I felt Chris captured in the following quote.

To me, it is worth using markdown because [team members comfortable with rST] are almost always also familiar with markdown, and thus it makes the documentation source more accessible and editable to a larger group. This is particularly true for outside committers, as in my experience the vast (vast, vast) majority of users don't know rST and it is intimidating to them to contribute to docs because of it.

My aim with this topic was to help us explore our thoughts about this together, so that those willing to put in work to change documentation tech choices know how we think in the team generally about this. I personally would like to put in work to make the repo's I work often in become Markdown+MyST repos, but not if it is a controversial change in itself within the team.

[rkdarst]

This is a good question which I think I shouldn't be involved in. But I have worked at the interface of both, and had to deal with this question with them.

I think that in the past, markdown and rst handled two different types of things: markdown mapped to html tags, while rst (in docutils/sphinx) was more like semantic LaTeX markup. And then markdown isn't standardized for semantic markup (and can't be because the original specifier doesn't want to use the name for a specification), so that.

myst_parser is great, and allows semantic markup in markdown, and hopefully is used more and more. But the difficulties in rst aren't about syntax, it's about how to use directives, semantic references to documents and anchors, etc. But learning how to do this in myst_parser is (to me) almost as much as learning how to do it in rst.

By the way, does myst have editor support? It is different enough from standard non-standard markdown that interpreting it as markdown doesn't give you all you need.

I'm sure myst_parser will win in the long term anyway, so I think there's not much for need for my thoughts on the comparison. But, we should prepare to learn almost as much new semantics with either.

[choldgraf]

does myst have editor support?

There is, though it's not standard across editors. There's an LSP implementation of the MyST markdown language: https://github.com/executablebooks/myst-language-support

and a VSCode extension for this here: https://marketplace.visualstudio.com/items?itemName=ExecutableBookProject.myst-highlight

Whenever JupyterLab also supports LSP I hope that it will make its way in there as well.

But the broader point is important to note: MyST Markdown is still young and the ecosystem of tools around it is still developing.

[minrk]

I haven't contributed to myst-based docs enough to speak from experience, but I understand that it lets us write markdown while still being able to use any/all directives when needed.

My general preference has been for markdown most of the time, and rst when the directives, etc. become more important (e.g. API docs or literal-includes, etc.). If myst had been around longer, I imagine the number of docs where a directive or cross-reference here and there caused me to switch to rst would definitely be much lower.

I do agree with others here that widespread conversion / migration / strict policy from rst to myst is likely not where I use of resources.

So my current inclination would be:

1. make myst available in general so that existing/new markdown docs may add directives here and there without needing to be converted to rst for that one .. literalinclude (is there a downside to using myst for pure markdown?)
2. take no other global or strict action
3. continue to write most new docs in markdown, which I think has been our convention. Converting a doc here and there while making major edits as @choldgraf mentioned is fine. Adopting myst should make this easier, I would think.
4. keep directive-heavy docs like these API docs as rst, even if/when we switch to myst most places. This is in part because I imagine the googleability of "how do I change how I use this directive" will be easier in rst, since folks won't have to look at rst examples and then translate those to myst in their head.

I think that my biggest challenge as a maintainer trying to be welcoming to new contributions is that I view the lack of a strict policy as a welcoming position. At the same time, some folks who are less comfortable in a given context can find an absence of guidance to be more of a room full of hidden spikes rather than the absence of them. This balance of "don't be too strict" vs "help people feel confident by giving them steps to follow" is hard for me. Of course, guidelines/tutorials for "a good way" to contribute don't mean they have to be "the way" to do it.

[willingc]

A bit more context: I use Myst in place of recommonmark and think it's great. We really should be using on all the Sphinx repos.

Encourage users to contribute changes to docs in the source format (md or rst). If greater than say 70% of the content would change, consider changing to markdown if the file isn't using intersphinx or custom directives or autodoc. If a file is converted to markdown, pandoc should be used and committed with a second commit that includes changed content.

This has been my approach for about 6 months and it has worked really well.

As myst continues to improve which I certainly hope it does, I may feel like conversion makes sense. For now, I would dump recommonmark for myst.

[choldgraf]

This https://github.com/jupyterhub/team-compass/issues/350#issuecomment-719706955 is basically what I'm doing as well 👍

(though one note - there is now an {eval-rst} directive so you can use that for autodoc :tada:)

[consideRatio]

If a file is converted to markdown, pandoc should be used and committed with a second commit that includes changed content.

Thanks for the guidance @choldgraf @willingc! I have never used pandoc before and just tried it out.

# step 1 - file conversion + commit
pandoc mydoc.md --from markdown --to rst --output mydoc.rst
rm mydoc.md
git add mydoc.md mydoc.rst
git commit -m "pandoc file format conversion: from Markdown to rST!"

# step 2 - modify new format
# (make edits)
git add mydoc.rst
git commit -m "Document effective climate change mitigation strategies"

Is this in alignment with your practices when you convert files in git version control @willingc and @choldgraf ?

[choldgraf]

@consideRatio in general yeah, though I am not doing this so much right now as I'm waiting to see if https://github.com/executablebooks/sphinx-tomyst will just make this very easy (since pandoc conversion isn't quite perfect)

[rkdarst]

Does newer pandoc handle roles and directives well? I try this:

:ref:`something <ref-target>`, to :doc:`some-doc`.

.. note::

   This is a note

and pandoc converts it to:

:ref:``something <ref-target>``, to :doc:``some-doc``.

.. note::

This is a note

But the myst equivalent should be:

{ref}`something <ref-target>`, to {doc}`some-doc`.

```{note}
This is a note

... which I think is a good example of how normal markdown concepts is not enough to express the semantics of advanced docs, and why I think using myst well also requires thought.

[update: yes, this is probably rhetorical since pandoc probably can't yet know about myst... because markdown isn't standard]

[willingc]

In general with pandoc @rkdarst, you will always need to do a visual scan of the output file. For most files, it will get you 90-95% there. The myst equivalents have worked very well. I wish they were around 5 years ago.

[consideRatio]

Hey all!

I'm currently considering championing work to transition the-littlest-jupyterhub and nativeauthenticator to MyST from rST. Before doing so, I'd like to understand a bit how you all feel about MyST at this point in time.

I figured I could raise some more specific questions and besides asking you to comment, I'd ask you to indicate if you are positive/neutral/skeptical about some statements.

:+1: = Positive :eyes: = Neutral :confused: = Skeptical

[consideRatio]

Do you consider MyST a good default choice for any new projects in the JupyterHub org?

:+1:=Positive --- :eyes:=Neutral --- :confused:=Skeptical

[consideRatio]

Do you appreciate seeing a JupyterHub project's documentation system transition to use MyST instead of rST?

:+1:=Positive --- :eyes:=Neutral --- :confused:=Skeptical

[consideRatio]

Closing as answered!