Planning: Markdown support

[davidtheclark]

There are two kinds of Markdown support to consider: Markdown embedded in JS, and Markdown files used as pages.

Markdown embedded in JS

Looking something like this:

import md from 'some-module-name';

const text = md(`Some **markdown**`).

This would provide the flexibility we'd want to be able to use Markdown anywhere that we have extended prose.

However, we don't want to bundle a Markdown parser and send it to the browser. I think the way to do this is to build a Babel plugin that uses remark-react at compile time to transform all uses of md().

Here's an existing Babel plugin to learn from: https://github.com/threepointone/markdown-in-js

Markdown files used as pages

This is the class Jekyll format. Both Phenomic and Gatsby support it, so it should be easy to learn from their open-source code. We'd use front matter to pass props to a specified layout component.

There's a big limitation with this format: Although (I think) you could embed HTML directly into the Markdown in a document like this, you could not embed JSX and use React components. If you want to embed JSX, it makes more sense to avoid this additional level of abstraction and write the page as a (relatively simple) JS file. I think that's ok. If people need to do a minimal amount of custom styling interspersed in their prose, they can do that with HTML. If they need more, they can create a page component.

[cammace]

To my knowledge, Docbox implements it's own markdown parser which works fairly well (although not the easiest to read) https://github.com/tmcw/docbox/blob/master/src/components/content.js#L14.

Although (I think) you could embed HTML directly into the Markdown in a document like this

I've been including HTML inside the markdown files inside /android-docs with no issues, see the Getting Started doc for example. MarkdownIt ignores any HTML syntax when parsing the markdown files. While this works, it would be great to reduce repetition and instead, create a React component and be able to include it inside the markdown directly. Creating a page component isn't ideal in my opinion since it isn't as readable and generally consumes more time maintaining.

Another requirement for parsing the Markdown would be to allow for variables inside the docs, similar to Liquid or Handlebars. This would allow for single line changes to SDK versions. I hacked my way through this by just replacing strings if they match the keyword. This might also be a solution for including react components directly inside the markdown?

[danswick]

Per @cammace's comment, should templating be its own planning issue?

[davidtheclark]

@danswick: Can you say more about what you mean by "templating"?

[cammace]

I think @danswick's referring to the Liquid syntax Jekyll offers, https://jekyllrb.com/docs/templates/

[danswick]

@davidtheclark: I was thinking something similar to Jekyll's approach to working with template layouts and includes, as @cammace said above. I suspect much of this will be taken care of with components, but I'm not totally clear on how components will work as a replacement. Is the goal of Batfish to replicate the functionality of tools like Jekyll to let authors define page templates ahead of time, then specify which of those templates should be applied to the content they write?

[davidtheclark]

Yesterday I spent a bunch of time investigating Markdown possibilities. I'll write down some thoughts before they vanish into the ether.

I'm seeing 3 different ways we could support Markdown. Maybe we should try all 3.

Markdown documents without JSX or expressions, with front-matter

This is easy to do. This is what Gatsby provides (and Phonemic). It involves a very short and simple Webpack loader — we could even use Gatsby's.

Advantages

• Super easy.
• Markdown documents are straightforward and simple — no templating, no nonsense, all prose business.

Disadvantages

• Super limited. I'm not sure if it addresses enough of our Markdown needs to be very useful.

Because this would be so easy to implement, we might consider implementing it in addition to other techniques that allow for more complications in the Markdown.

Compile-time Markdown component

A JSX Markdown component that takes effect at compile time, as a Babel plugin, instead of runtime. Use it to wrap markdown, either as blocks or inline. babel-plugin-jsx-markdown, we could call it.

Examples:

<Markdown>
  ## Heading 2

  Lorem ipsum **dolor** sit amet, consectetur [adipisicing](#thing) elit.

  Another paragraph.
</Markdown>

<p>
  Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do
  <Markdown inline={true}>labore _et_ dolore</Markdown> magna aliqua.
</p>

Advantages:

• Very flexible and embedded. Widely useful.
• Encourages authorship in JS documents, which means unlimited power, good debugging, no additional magic.

Disadvantages:

• Not sure how feasible it would be to be able to interpolate expressions within the body, e.g. <Markdown>you have {hairCount} hairs</Markdown>.
• Not sure how feasible it would be insert JSX into the Markdown component body: instead, you may need to intersperse Markdown and other JSX elements.
• Not a document format: works within React components in JS files, not in special .md files.

Prior art:

• Surprisingly, I can't find anything that is trying to do this, specifically.

Challenges:

• I need to get better at writing Babel plugins.
• Allowing the interpolation of expressions and JSX might prove pretty difficult.

Enhanced Markdown documents with JS expressions and JSX

Much like Jekyll's Markdown documents, except instead of Liquid you could us JS, and instead of HTML you could insert JSX.

Here's an ideal example:

---
title: My page
description: 'I made it'
version: 4
import:
  WonderfulComponent: "../../wc"
---

## Introduction

This is **my page.** It uses Markdown.

You are looking at version {{ version }}.

<div className="mt24">
  Here's some JSX.
</div>

<WonderfulComponent works={true} />

Advantages:

• Close resemblance to the Jekyll Markdown documents everybody is used to.
• Great power.

Disadvantages:

• Same as above: Allowing the interpolation of expressions and JSX might prove pretty difficult.
• Encourages authorship in non-JS documents, which is more magic, harder to debug.

Prior art:

• https://github.com/ticky/markdown-component-loader is essentially aiming to do this. It looks pretty good, too, though not super flexible or configurable at the moment. Also, it's weird to me that it operates with regular expressions instead of JS parsing — a little bit concerning, given how difficult that probably is to get right. So I could try reaching out to the author to ask about these concerns.

[KyleAMathews]

On compile-time Markdown, there is https://github.com/threepointone/markdown-in-js

[davidtheclark]

I spent some time experimenting with a compile-time Markdown component. markdown-in-js is so complicated and overreaching that I'm not sure it'll be an option for us. I came out with this experiment: https://github.com/davidtheclark/babel-plugin-transform-markdown-in-jsx. The big caveat, maybe crippling for it, is that code in the Markdown (inline or block) is very dangerous, because curly braces will mess up the JSX and most code uses curly braces. The way out of that might be to suggest using a React component that accepts strings to display code.

However, the more I've thought about it, the less important I think a compile-time Markdown is. If you're writing small chunks of Markdown embedded in JSX, you could just as well write straight JSX most of the time. The real value of Markdown parsing is in Markdown documents for highly templated pages. markdown-component-loader looks good for this, and the author is active and responsive, so I think I'm going to dig deeper into that.

[davidtheclark]

Not happy with the way babel-plugin-transform-markdown-in-jsx is going ... too many kind of confusing caveats. I'm leaning towards https://github.com/davidtheclark/md-jsx-transform as the solution.

[davidtheclark]

Somewhat related to this, in the sense of creating tools that help writers of pages without adding to the page bundle ... I learned more about Babel plugins last night by working on https://github.com/davidtheclark/babel-plugin-transform-syntax-highlight. I think this is working well, and code be a good way to get syntax highlighting when you're not using a Markdown document.

[davidtheclark]

I made https://github.com/mapbox/md-react-transformer and a Webpack loader for it, https://github.com/mapbox/md-react-transformer-loader.

Next week I'll try to incorporate the loader into Batfish.

[davidtheclark]

Markdown support is in, with https://github.com/mapbox/jsxtreme-markdown.