Move from Markdown to RST?

[giovannipizzi]

As discussed in this comment by @rartino, managing numbering of sections is becoming a big burden. Should we move so some different language (e.g. RST that supports automatic numbering and much more?) There is probably a small barrier for users that are not used to RST but I think it's probably ok?

[merkys]

RST would allow us to handle cross references easier, right? If so, I am in favor of this change.

[giovannipizzi]

RST would allow us to handle cross references easier, right?

Yes

[dwinston]

I would like to bring attention to MkDocs as an alternative solution to the problem of section link management.

Pros:

• still Markdown
• still a static site (i.e., hostable by github pages. convenient mkdocs gh-deploy command.)
• sections can be broken into separate files for fewer merge conflicts, without sacrificing document search (built-in search with context-highlighted matches)

Cons:

• Rendered markdown may differ from github preview. Recommended to mkdocs serve and preview in local browser before git commit and push.

[rartino]

Do I understand correctly that with an MkDocs-based solution there would be no way to see a rendered file with section numbers and cross references inside github? (I assume that we do not want to check in a rendered copy, e.g., generated by a hook, since that would trigger unnecessary conflicts?)

[dwinston]

With a MkDocs-based solution, one would not have the convenience of the built-in Github website editor for instant feedback, i.e. editing a Markdown file and clicking a tab to render and preview your changes. Instead, one would need to edit that Markdown file locally and check the rendered preview at e.g. http://localhost:8000 (regenerated automatically on file changes).

In any case, we would not check in a rendered copy. The typical flow is to check in a rendered copy only to the branch that serves the website. For Github Pages sites (https://<org>.github.io/<repo>) this is typically the gh-pages branch, but in our case we set it to the master branch. The idea is that the develop branch never gets rendered html checked in; rather, it serves as source for the html output built to the website-serving branch. Interestingly, I notice that https://materials-consortia.github.io/OPTiMaDe/ redirects to http://www.optimade.org/OPTiMaDe/, which displays differently than http://www.optimade.org/optimade because URL PATHs (the part after /) are case-sensitive, and only the former URL properly renders the "theme" indicated by the _config.yml in our master branch!

This means also that it is non-trivial to review rendered output on github pertaining to a pull request, as github preview will use its default markdown renderer rather than use MkDocs. Thus, a reviewer needs to check out the branch of interest locally and mkdocs serve to preview the rendered output. In my group, I set up a server that uses github commit hooks to re-render previews online for each branch, but this of course is a custom solution.

[rartino]

A quick note: in #131 it was noted that it would be nice if there was a way to identify filter string examples so that they could be tested automatically for syntax. RST has syntax for this. One-liners can be given text roles, e.g.

:optimade-filter:`identifier HAS EXACTLY values` 

[sauliusg]

Having code one-liners marked would be a valuable feature.

What about multi-line code examples, such as JSON response examples?

Does GitHub render RST?

[giovannipizzi]

Does GitHub render RST?

Yes, it does, see e.g. here. However note that RST documents are intended as pages of larger documents, and moreover can be extended so rendering of a single page is not perfect. If we go to RST (that I would personally support for the advantages it give) we will probably need to have also a compiled version somewhere (this can be automatically compiled by e.g. Travis and deployed on the OPTiMaDe website, for instance). The disadvantage of RST is that you need this compilation step (but for such a small document it's going to be very fast) and that the syntax is a bit more verbose (but much more powerful).

[sauliusg]

Thanks, @giovannipizzi , for the link!

But as I see there, the tags are rendered in the final text. If so, we can do that in MD just as easily:

filter: identifier HAS EXACTLY "string value"

I was thinking to propose such solution, but though it was to hackerish to be acceptable for the rendered documentation. But if the tags are visible anyway, why not tagging them just in MD?

[giovannipizzi]

As I said, not everything can be rendered by GitHub. The tags are extensions. In a compiled documentation (e.g. with sphinx), they will not be visible but e.g. point to the right link, or be properly rendered. See for instance the compiled version of that document here.

Note also that if we use readthedocs to render the compiled version, beside being automatic on commit, it will also allow other interesting features (maintaining multiple versions of the docs, which I think is a very big plus; having also a PDF version; ...)

[sauliusg]

At the very beginning, the suggestion was to write the specs in LaTeX, which would have all the features one would wish and some more. AFAIR, the suggestion was declined precisely because LaTeX requires a compilation step.

Is that attitude changing now? If yes, we could consider other interesting possibilities, like:

• LaTeX (https://www.tug.org/texshowcase/);
• DocBook (https://tdg.docbook.org/tdg/5.2/);

Cheers, Saulius

[rartino]

While it is true that not every feature in RST renders correctly in github, a lot do. The feature set that renders correctly is certainly on parity with markdown PLUS automatic table of contents, auto-numbered sections, auto-numbered footnotes, etc. which will significantly decrease the workload of dealing with merge conflicts.

The weird un-rendered links, etc., in @giovannipizzi's example is because they use sphinx-specific syntax. With normal RST, the corresponding features render correctly for me. Here is an example I have composed to show off features we need for OPTiMaDe: https://github.com/rartino/OPTiMaDe/blob/restructuredtext/demo.rst (as usual, click 'raw' to see the source document.) Note how also the optimade-filter-tagged one-liner is rendered correctly.

A few points in favor of RST:

• Correct auto-rendering inside github is a huge plus. Being able to look at github's "rich diff" of a commit by a simple click is worth a lot compared to having to clone the repo and run a local renderer, or fiddle with checked-in pre-rendered versions.

• I really like these lightweight markup formats (MD, RST) over markup-heavy alternatives (e.g., LaTeX) because they also make regular diffs easier to read.

• A benefit of RST over MD is that even quite basic features of MD requires embedding html code, e.g., image formatting, anchor links. I wouldn't call that output-agnostic, whereas with RST one can realistically aim for a specification document that can be rendered across output formats without code that targets each format.

Are there any arguments against RST? So far I've only seen "it isn't markdown". There is some merit to that; I assume markdown is the most well-known of the formats with lightweight markup. But RST syntax seems as close as one can get to markdown without actually being markdown.

[giovannipizzi]

I also support RST ovee the other suggestions for the reasons discussed. Moreover, it's more lightweight, it can be rendered nicely to HTML (also latex can in principle, but the output isn't particularly nice), and RST can instead be converted automatically to latex. And I also agree that direct rendering in the browser is a big plus.

[sauliusg]

Hi, folks,

the ability to tag code with specific names seems valuable; to me this would be an argument for RST.

Is the an md2rst converter?

On 2019-06-21 02:12, Rickard Armiento wrote:

The weird un-rendered links, etc., in @giovannipizzi https://github.com/giovannipizzi's example is because they use sphinx-specific syntax. With normal RST, the corresponding features render correctly for me. Here is an example I have composed to show off features we need for OPTiMaDe: https://github.com/rartino/OPTiMaDe/blob/restructuredtext/demo.rst

Sorry for a homework-not-done question (honestly do not have time to dig into RST docs at the moment), but why is aiida_core's:

:ref:calculation functions<concepts_calcfunctions>

rendered differently from optimade's:

:json:{'hello':'data', 'cucumber':42}

I don't see the reason why the first is rendered with :ref: on GitHub an the other without the :json:? Does it depend on different project setup, or is there another reason?

Regards, Saulius

-- Dr. Saulius Gražulis Vilnius University Institute of Biotechnology, Saulėtekio al. 7 LT-10257 Vilnius, Lietuva (Lithuania) fax: (+370-5)-2234367 / phone (office): (+370-5)-2234353 mobile: (+370-684)-49802, (+370-614)-36366

[merkys]

I suggest waiting until freeze takes place (according to the schedule, today), and then opening a branch to migrate from the MD to the RST. Unless, of course, someone presents a reason not to.

[dwinston]

Thank you for the demo page, @rartino. It's clear to me that RST has the advantage of auto-github-preview while eliminating the pesky manual table-of-contents updating. I also agree with @merkys that migration to RST should not block a spec freeze / stamp-as-version.

[rartino]

why is aiida_core's: :ref:calculation functions<concepts_calcfunctions> rendered differently from optimade's: :json:{'hello':'data', 'cucumber':42}

In the demo document I linked, :json: and :optimade-filter: "text roles" are defined as code-type literals for respective language in a segment at the top of the file (you can think of it as similar to a LaTeX document's preamble.) The sphinx software that heavily relies on RST has some extra text roles pre-defined, e.g., :ref: allows slightly more convenient cross-referencing. Since these text roles are not recognized by github's "raw" RST renderer, they render as unresolved.

Is the an md2rst converter?

I have used pandoc in the past. It isn't perfect, so the conversion job will likely take a couple of hours of manual cleanup.

I suggest waiting until freeze takes place (according to the schedule, today), and then opening a branch to migrate from the MD to the RST. Unless, of course, someone presents a reason not to.

My thinking was to release v0.10 in markdown, and right after that release do the switch in develop. That way no ones hanging PR should have to be converted from markdown to another format.