Interop with MyST / mdx

Hi @jdkato: first off, congrats on the awesome ideas and projects. I ran into markdata while browsing markup-related projects and couldn't help but notice the stunning similarity of the problem you are solving - and even in the approach! - with MyST, which is exactly what you describe, markdown extended with RST roles/directives (and so also with includes etc), which is "the good part" of rst. As you write, rst's problem is lack of native parsers and relatively smaller adoption than md, something I'd also always bemoaned. That's why I was super excited to see MyST which plugs this extended md into the Sphinx ecosystem, which is a gem in itself.

So basically they do:

```{directive}
:option: value

content
```

and role{something} - and there you have it, directives and roles. It's actually so clever in its simplicity.

The project is coming from the Jupyter Notebook community, which is another nice thing - interop with Jupyter is also on their radar. So this is in a sense already joining a bunch of markup use cases that didn't necessarily play well before. Loved it when I saw it (just like I love your solution BTW, what's not to like about Python ;))

Now in the so-called meantime, another interesting phenomenon happened - mdxjs and React/custom elements in general. While it always had the "oh I wish we could have that interactive awesomeness in the rst world" effect on me, and I'd been looking with interest at things like docusaurus - I never really thought there was any way these two worlds could be easily connected. Except when I saw the MyST solution of "take md and just drop in directives and roles into it" I thought - hey, the way this works is actually pretty darn close to - at least a subset of - mdx, which takes markdown and enables you to add custom elements. Now - element, role, directive: who cares what it's called? Effectively it's the same thing, extending the markup syntax with custom, potentially user-defined, stuff, either in-line or block.

So I went on and created this issue which as you can see would add a flavor of MyST that simply uses custom elements to create directives and roles. I was curious to see MyST's response, which was actually very positive (another great feature of their community). Anyway, that kid of approach would mean that any valid MyST document in that syntactic flavor would also be a valid mdx document. I don't mean full javascript capability - that is I think going too far - but the custom elements are such an obvious match to directives/roles (and attributes are trivial, since HTML has attributes). So this kind of mdx-myst document would be possible to run through a Sphinx documentation flow while simultaneously being open to implementing custom elements in JS to cover the mdx use case. I do believe there would be still work to do in the JS world to get support for some of the basic e.g. Sphinx directives more "built-in" to default JS markdown parsers, but most of the work is done for us already, and the fallback for unknown custom elements is quite OK - the thing just gets displayed as text.

I chanced upon markdata just today, but essentially you'd implemented a very similar kind of thing, just with a Python preprocessor. Simple and elegant too, but requires a preprocessor! (and also adopting a new syntax) Thus I ask - perhaps we could join forces here and plug entirely into the mdx/MyST ecosystem (both of which have a solid user base)? Essentially you could do exactly what you're doing, but use mdx as input. Tables - perhaps just use the Sphinx/rst table capabilities, perhaps even we should extend Sphinx to have a data-table capability that takes in not only CSV, but also JSON and YAML, that's probably an implementation detail along the way (I'd love to see that too some day).

The challenge with a new (even slightly different) syntax is that in every project/documentation you have to decide on a syntax which effectively means you 'fall' into some specific ecosystem, but if we make them ALL compatible, you won't have to decide that much any more ;)

jdkato / markdata

Interop with MyST / mdx #3