[Proposal] Move to CommonMark via Markdown-it instead of Showdown

[callahad]

"Markdown is the heart of Ghost, and as such we intend to make working with it one of our core competencies." — Ghost Wiki: "Future of Markdown"

Ghost wanted a spec. CommonMark is it.

If markup is one of Ghost's core competencies, then that markup must be rigorously specified. Markdown is notoriously ambiguous and poorly defined. Without a well-defined spec, other tools can't reliably integrate with Ghost. The Ghost wiki explicitly calls out the need for a spec, stating that "the goal is to create something like: https://github.com/mustache/spec"

Fortunately, in the year since that page was last edited, the CommonMark specification has emerged, matured, and stabilized. Much like HTML5, CommonMark made compatibility with existing documents an explicit goal, and wrote its spec with an awareness of common behaviors in existing Markdown parsers. The resulting spec represents three years of work and fits Ghost's needs. it would be foolish to duplicate that effort or further fragment basic Markdown syntax.

Embracing CommonMark gives Ghost a well-defined foundation with a rigorous, automated test suite. Ghost-specific extensions could be concisely defined in relation to CommonMark's baseline.

Markdown-it is the right library.

The two major JavaScript implementations of CommonMark are the reference implementation and Markdown-it. Markdown-it is the better choice for Ghost:

• It's been actively maintained by a pair of developers for over a year.
• It can optionally be configured for strict CommonMark compliance.
• It has a stable, plugin-based API for syntax extensions.

Plus, Markdown-it is fast and already ships with plugins for things like GFM-style tables and strikethrough, among others.

This solves real problems.

In addition to resolving ambiguities in Ghost's markup, switching to CommonMark via Markdown-it would make it easy to address:

• Smart typography (em-dashes, curly quotes, etc.): #534, #1795, #4649, UV#7196701
• Table support: #1670, #4777, UV#7312035
• Dialect confusion / ambiguity / extensions: #1642, #2675, #5077, UV#7675026
• Specific requests for CommonMark: #3959, UV#7196681

This would also go a long way in alleviating problems with maintainability:

• All of the problems in #2381, "Make the showdown extensions make sense"
• "Ultimately this is a problem with the showdown library rather than with Ghost. [...] Showdown is unfortunately quite buggy." — @ErisDS on #2675
• "This is one of the many features that would be an absolute nightmare to crowbar into the existing showdown markdown conversion system." — @ErisDS on #4649.

Thanks to Markdown-it's API, it's easy to create small, clean, and independent plugins for Ghost as needed, especially compared to maintaining an entire fork of Showdown.

Ghost can get there incrementally.

Everything is in place for a graceful transition:

• Rendered HTML gets stored in the database, so existing posts are safe by default.
• Showdown and Markdown-it can both exist in package.json at the same time.
• Markdown-it can live behind a Labs setting for as long as needed.

If Ghost commits to moving to CommonMark + extensions, it would have to figure out what to do when users edit posts that predate the CommonMark transition. By order of increasing effort, Ghost could:

1. Show a generic warning on old posts.
2. Produce a rich diff by running old posts through both Markdown-it and Showdown, or just comparing Markdown-it's output to the HTML in the database.
3. Offer a "quirks mode" that sends posts through Showdown instead of Markdown-it.
4. Offer a "quirks mode" that uses custom Markdown-it extensions to mimic the idiosyncrasies of Showdown.

In all cases, the end goal would be to eventually drop support for Showdown's quirks. Ghost Foundation employees could run the entire Ghost(Pro) corpus through both Markdown-it and Showdown to measure and mitigate any potential breakage. Similarly, someone could run the CommonMark test suite through Showdown to help understand fundamental differences between the two engines.

Again, since rendered HTML is in the database, there would be no risk to existing, published posts.

TL;DR:

I really want curly quotes.

[superlou]

Would this make it possible to disable markdown formatting based on custom delimiters, e.g., LaTeX script inside "$"s or "$$"s? Escaping underscores to prevent italicizing continuously in LaTeX is tedious.

[7k50]

Markdown table support would be fantastic.

[smscotten]

At the risk of making a "me too" post, I'd like to speak to the larger point. While there are specific features (tables) holding up my migration to Ghost from a Drupal blog where my posts are already in Markdown, it's really the larger scope that matters here. Markdown may be best when there isn't a universal standard, but having a featureset somewhat consistent with the authoring tools out there (editors like Byword etc) is really important to the ability to get something written and out there.

But also, yes, I really do need the tables. For the moment as I'm doing the migration work I'm just making sure the spacing is right and wrapping my tables up as codeblocks. Yay for Markdown being readable even as plain text, but I have reservations about relaunching with tables in posts looking that way.

[nybblr]

I would love to be a part of this transition—the showdown fork has been my main barrier to entry in the codebase, but I would be incredibly excited to see Ghost move to CommonMark so I can keep preaching "One True Markdown" (said with slight troll accent).

[kashev]

Currently Commonmark doesn't have footnotes, so switching fully would cause some people to lose functionality.

[nybblr]

There's a pretty slick demo that includes the footnotes plugin: https://markdown-it.github.io/

[nybblr]

Just to take this for a spin, I've swapped out the markdown engine on my fork: https://github.com/TryGhost/Ghost/pull/6352

Would love to tease this out first part of the process!

[nybblr]

This is looking really feasible. After the swap out, I'm down to 4 failing specs (all related to the image placeholder, which I'm writing a plugin for).

https://github.com/TryGhost/Ghost/pull/6352

:eyes: on the PR would be :+1:

[lu-zero]

@nybblr is the set on hold for the editor changes or you plan to keep maintaining it?

[evil-shrike]

I used Markdown-it as MD engine in some my static site generator, it's a real fun. It has very flexible plugin system. It'd be cool to have it in Ghost. Is there any progress in this space?

[ErisDS]

As we're replacing our editor with a mobiledoc editor, this issue is going to be less relevant. We're planning to stick with showdown for now for the markdown functionality, to ease the transition pain.

I'm going to close this for now, however it can still be voted on here: http://ideas.ghost.org/forums/285309-wishlist/suggestions/7196681-commonmark

[sandorfr]

Well we can't vote anymore on the link.... Very sad to hear you won't do anything about it and even worse move to a wysiwyg editor :( .

[kevinansfield]

@sandorfr Ghost is using markdown-it/CommonMark since v1.0

[sandorfr]

oh :) that's a relief. Some upgrade to do then. I hope the mobiledoc thing was a bad joke or is it still planned? plus you should change the uservoice tag to something else than declined!

[kevinansfield]

@sandorfr the mobiledoc based editor is definitely still planned, you're free to have a single markdown card in each post if you only want to use markdown though - that's what's done in 1.0 at the moment

[corazza]

I was really confused because I understood MobileDoc was coming with 1.0, that this was the main thing. But I couldn't see the difference. So, when is it actually planned?

[sandorfr]

Thanks for the clarification @kevinansfield :)