Write out a strategy / vision for MyST markdown

[choldgraf]

Description

In #526 we discussed how we'd like to discuss the vision for MyST Markdown as a community standard for authoring scientific content. We agreed that it'd be useful to write this up as a short "strategy" document and post it in the executablebooks.org repository as a blog post.

Notes from the meeting

Here are a few rough notes about ideas we had discussed, we could take these and combine them with some other high-level strategy language for the post.

• Un-tie MyST from the Sphinx ecosystem so it is clearer how it could be re-implemented elsewhere
  • Define the syntax in an implementation-agnostic way
  • What would the intermediate format be?
  • markdown-it tokens?
  • Some kind of AST (ideally not Sphinx!)
  • Does MyST need to be multi-document? (cross-refs etc)
  • No, general agreement that some other application should deal with things like refs
  • What is the functionality that you'd need to enforce across implementations of MyST, vs. what can be implementation-specific.
  • A good design case is a MyST Previewer in something like VSCode - this could be a starting point for minimal syntax and directives that a myst parser must support.
  • Graceful degradation for syntax that isn't part of the core-spec
• Vision for extending into other ecosystems (like prosemirror, pandoc, word, latex papers, etc.)

[choldgraf]

Assigning this one to @rowanc1 since he agreed to take a first pass at this document!

[rowanc1]

I have started by putting a few issues around the place, specifically in [https://github.com/executablebooks/meta/discussions/662] (which I just learned that discussions don't add backlinks 😢 ).

A bit of the things that come to mind on this ticket are below. I will start to formalize some of these into the documentation and writing that I am doing while going through mystjs and thinking how some of the other tools we have recently built openly fit into this (latex templates, jtex).

---

Together with @fwkoch and @chrisjsewell we have been working on an AST that is an extension of mdast. As part of that I have dived into the unified community, as well as MDX to see what we can do to help there. I think the short answer is: an extensible, well documented markdown language is needed in many places. MDX solves parts of this, but not many standard pieces, and I have yet to see something that is as comprehensive as MyST is now and can become in the future (with the appropriate enhancement proposals in place!).

I have also just gone through all of the discussion items on the meta repo, and many people are currently coming to EB from bookdown and are excited by the community activity, the possibility of improvements over time, and the connection to jupyter. There are also people coming with the expectation that "publication quality" means a latex pdf or my thesis, so I think that some of the expectation of integration to a wider ecosystem is already in place.

I have also been thinking hard about how much of the pandoc ecosystem we want to duplicate. There are decades worth of effort in that tool. I can think more on why to take on some of the responsibilities here, but top of mind are: (1) a javascript and python implementation opens up many doors (e.g. vscode extensions, in browser execution, language servers, new contributors, new communities); (2) a focus on computational & interactive content (e.g. thebe, inside jupyter); and (3) modern, not-yet-invented, web-publishing frameworks for scholarly communication (e.g. nueolibre, joss, curvenote, distillpub etc. are hinting in this direction). Two other things come to mind: (1) the javascript community has largely ignored pandoc, and has invented many other ways to do similar tools and conversions, meaning this task is more stitching together/bridging, rather than starting from scratch; and (2) we can make a bridge to and from myst <--> pandoc md to gain access to all tools in the pandoc ecosystem.

I think that there are three important areas to focus on beyond sphinx/rst for myst:

1. web and html (including modern thesis/paper formats, e.g. distil, others)
2. pdf & latex (yes, they are still important)
3. word (yes, still important)

I think all of these are so very close, and many of the other formats (epub, jats) we can use pandoc for.

[choldgraf]

Update

We discussed some of this in our team meeting today, and had a few related points that I'll put below:

• Make MyST a top-level project. We agreed that MyST Markdown deserved to be a top-level project alongside Jupyter Book, as opposed to a child of Jupyter Book
• Centralize MyST docs on a single domain. We agreed to have a landing page website for MyST Markdown, and use sub-domains to house more in-depth documentation for it. Both myst.org and myst.com were taken, so we decided to go with myst.tools as a start. Here is a rough idea for the domains:
  • myst.tools -> a landing page with some nice demos, high level info about the project, looks nice and splashy, etc.
  • spec.myst.tools -> more thorough documentation about the myst specification
  • js.myst.tools -> points to the mystjh docs
  • sphinx.myst.tools / python.myst.tools / ??? -> points to the Python myst-parser implementation (not sure about this one)
• Standardize our spec docs in a single place - this should be one repository that is the "source of truth" for MyST Markdown, where we define the spec, have a comprehensive list of tokens and how they look when rendered, and give guidance for others that want to build on top of it.
• Define more explicit governance around MyST Markdown - if we want to formalize MyST as a project, and attract other community members, we'll need more specific governance and policies for decision-making. We have a starting point in the extension proposals repo but we should agree on a specific process etc as a group.

We'll need to implement these pieces in the coming months, but wanted to note the decisions here for now!