Migrate from hoedown to pulldown-cmark for markdown parsing

[frewsxcv]

For markdown parsing, we currently use rust-lang/hoedown (based on hoedown/hoedown). There's a (somewhat) new Markdown pull parser entirely written in Rust: google/pulldown-cmark. Some reasons to switch:

• All the safety guarantees that come with anything written in Rust
• Rust contributors don't need to know C if they want to make changes upstream
• Don't need to maintain our own C hoedown bindings / pulldown-cmark has an idiomatic Rust API
• It uses Cargo, which should make integration easy

[frewsxcv]

cc @rust-lang/tools @rust-lang/docs

@GuillaumeGomez has already expressed interest in making the changes for this transition.

[frewsxcv]

Via https://github.com/rust-lang/rust/issues/29329 and https://github.com/critiqjo/rustdoc/pull/2, it looks like @critiqjo has already made some progress on migrating to pulldown-cmark.

[alexcrichton]

We're not quite ready to pull in crates.io dependencies into the build system yet as we've still got makefiles that don't work with that. On 2016-02-02, however, I'll be deleting the makefiles with glee at which point we are capable of pulling in dependencies from crates.io!

[frewsxcv]

2016-02-02

I'm going to guess you meant 2017 here

[alexcrichton]

Oops, yes!

[steveklabnik]

I have been very interested in this for a long time.

[matklad]

Some random thoughts:

1) Are these two implementations compatible? Probably not, and that means some docs might break a little bit.

2) Perhaps this is the chance to specify what exactly is the markdown dialect of the docs?

3) What's the current plan about using ::foo::bar::baz syntax for cross-link the docs? If we decide to add some kind of linking, will it be compatible with pulldown-mark?

[frewsxcv]

Are these two implementations compatible? Probably not, and that means some docs might break a little bit.

pulldown-cmark is an implementation of CommonMark.

Hoedown follows the original markdown document, but there are talks about switching to CommonMark.

From what I understand, one of the main reason CommonMark exists is because the original Markdown specification is so vague and there were many subtle differences in implementations. More info in this thread. I'm not familiar enough with either spec to know specific differences.

What's the current plan about using::foo::bar::baz syntax for cross-link the docs? If we decide to add some kind of linking, will it be compatible with pulldown-mark?

This is what led me to open this issue. There aren't any concrete plans right now for this syntax, but if this something that is decided cmark-pulldown can help. There are talks about adding a basic generic directives/plugins syntax into CommonMark which would be great since we wouldn't need to invent our own syntax. Though, if we did invent our own syntax, from what I've seen of the pulldown-cmark API, it seems it would be easier to implement in that than Hoedown.

[critiqjo]

Can we move out the book-generating feature from rustdoc? Since a new book in the making, which is soon to be The Rust Book, and may be this is a good time to do some cleanups?

[steveklabnik]

Can we move out the book-generating feature from rustdoc?

There is no book generating feature in Rustdoc.

The current book uses a tool called "rustbook", and the new book uses "mdBook". Both are built on top of rustdoc's support for standalone markdown files, but that's it. I wouldn't want to move that out of rustdoc; as it'll be used in other places (like standalone documentation that's not book-like). It would also be a big breaking change, so that's not acceptable.

[critiqjo]

I see... I think I misinterpreted certain parts of librustdoc the last time went through the source (9 months ago) when I was refactoring it. I'll take a better look soon, and I've yet to look at rustbook.

[Ericson2314]

I see a src/vendor directory? Can it be used until that magic februrary date?

I've wanted to get rid of hoedown for like a year and half now, very excited this is imminent!

[frewsxcv]

FYI, @GuillaumeGomez has started work on this.

[frewsxcv]

Completed in https://github.com/rust-lang/rust/issues/40338 by @GuillaumeGomez! 🎊

There are a few remaining things to take care of after this migration: https://github.com/rust-lang/rust/issues/40912

For all intents and purposes, this issue is complete though