Sundown backend for rustdoc

[brson]

Sundown is the C markdown parser/renderer used by GitHub. We'll start by using an external markdown or pandoc, but eventually it might make sense to integrate a markdown library. Probably we would want to do this by creating an external rust-sundown cargo project, and either integrate cargo into the rust build or just move rustdoc out of the rust repo.

[graydon]

https://github.com/tanoku/sundown for reference. And yes, we should probably have a rustdoc mode that emits "finished" HTML renderings directly.

[thoughtpolice]

I'd be interested in working on this. I like the idea of rustdoc being distributed and updated by cargo. However, I have some other concerns that should probably be split off in another ticket (maybe one already exists?) I think it will be important as people begin writing more bindings for Rust.

The thing is, that it would be nice if rustdoc could just distribute the C files as part of the build, to minimize the need to build and install a dynamic library. However, I see no way of having Cargo handle this at the moment as it only builds crates - for comparison, GHC accomplishes this by just accepting C files on the command line and giving them to gcc appropriately as part of the build/link. So that's one thing (just give any c/cpp files to rustc as part of the build) I think has a pretty good power-weight ratio.

In general I tend to use this kind of feature a lot to bind libraries in a 'standalone' manner - many small libraries such as sundown, leveldb, etc. can be included in the source without much hassle and the need for external installation (and in the case of both sundown and leveldb, there is no install target in their build system anyway.) This makes the package easier to install from a users point of view and requires one less global library (and headaches installing it.)

Furthermore, this kind of support in the frontend/cargo makes it much easier to write rust libraries that have C wrappers for libraries that do exist on the system. If you have a C++ library like re2, which is installed in /usr/local or whatever, it's easy to just wrap up the basic functionality in a C shim expose it from rust. Then you can just give the shim to the compiler with your crate, and it'll build it and link it as part of the result. Alternatively, Cargo can handle this.

Finally, one point worth considering too is that, in GHC for example, C files you give it can include FFI headers and all sorts of stuff from the runtime system because it sets -I paths for gcc appropriately; arguably, you may not want the user to fumble around with finding header file installation paths if they do want to interface with rt, since the compiler being invoked will have direct knowledge of the correct path anyway.

In general I don't know what this means for native functions, since native modules depend on the name of the module/link_name to determine what shared library to use. The FFI in GHC merely lets you point a function to some arbitrary symbol name.

This problem can be attacked multiple ways. They all involve Cargo, really. It can handle all this logic of passing C shims to gcc directly, or rustc can handle it, and cargo just appropriately passes the .c files to rustc, etc. There are plenty of good reasons why putting this functionality in Cargo might be better, but I think just having rustc able to handle .c and .cpp files appropriately is a good push in the right direction and nails lots of easy cases like this without too much complexity gain. Support for this would make this ticket very easy, ultimately allow rustdoc to be handled by cargo (with an installation that can happen right after you install just rustc and cargo, since there aren't other external dependencies,) and it would buy a lot when people want to Cargo to build other kinds of bindings (that are standalone, need wrappers or interface with rt, etc.)

Sorry for this being long, it perhaps should be split up into multiple tickets. So please let me know if this is way off base, or sounds crazy, etc. Maybe nobody even wants to be able to do this, maybe it's already done and this ticket is easy? I find myself running into this case a lot, personally. If this seems like a good idea though, I'd be more the happy to make a ticket and begin hammering away.

[catamorphism]

@thoughtpolice Are you still interested in working on this? (Looks like #1850 blocks it.)

[thoughtpolice]

@catamorphism Yes, I am still interested in working on this. Sorry for the absence, I've had a lot of my spare cycles taken recently by a new job. #1850 does indeed block this issue quite heavily at the moment it seems, but @graydon proposed bundling clang with the LLVM repository that Rust uses in #2084. Another alternative is simply calling out to the system gcc. But those issue are left for those tickets, not this one, so I'll review all the comments made since I opened those bugs and talk about it there.

[catamorphism]

@thoughtpolice Great, I'll leave this open but consider it blocked for now.

[catamorphism]

This continues to be blocked on rustpkg being in a better state.

[thestinger]

There are many differences between the markdown dialects used by pandoc and github, so we would have to pick one. For example, both support syntax highlighting and tables but they don't agree on a syntax.

The "standard" markdown subset is very crippled so sticking to that isn't going to be enough. Pandoc is the nicest superset out there, but it's a very heavy dependency to have.

[pnkfelix]

I idly wonder if there is (or if one could make) a sundown backend for pandoc.

[emberian]

https://github.com/mozilla/rust/issues/8125

If/when we have a doc generator in Rust, it'd be worth reconsidering this issue.

[emberian]

The current consensus is that we're going to use pandoc. We'll have a pure-Rust, pandoc-compatible markdown renderer for fallback when pandoc isn't available. Closing.

(cc @Seldaek, @thestinger)

[thestinger]

See #9377 for follow-up.