Expand scope or new book for ecosystem examples

[lambda]

One of the places in which Rust by Example is not as comprehensive as it could be, as described in this recent Reddit thread on whether effort should go into examples on Stack Overflow Documentation, is in examples outside of the core language and standard library.

Now, there's pretty much an unlimited scope for what could go into example utilizing the full ecosystem. However, there are a number of prominent, widely used crates that almost anyone using Rust will likely want to use, depending on what they are doing; crates like libc or winapi for interacting in more depth with the platform, nix for higher-level platform features, mio and its higher level interfaces for async programming, regex for, well, regexes, iron or nickel or other web frameworks, piston for gamedev, gtk for UI, diesel for ORM, etc.

You could argue that each of these projects should have their own examples. And that is true; but that leaves a fairly inconsistent experience for learning once you expand past the standard library; some just have readmes with examples, some have dedicated docs sites, some don't have anything beyond the API docs. And some examples you might want to write cut across different projects; webdev examples, for instance, might want to demonstrate using iron and diesel, or compare how you would solve the same problem with iron, nickel, or rustful.

Since Rust by Example is already fairly large, it would probably be best to have a companion book, or maybe even a series of companion books, for ecosystem examples. Of course, the pretty much unlimited scope of this request brings up the question of how you decide when there is critical mass on a particular topic to actually start a new section of such a book or new book, versus relying on the existing ecosystem documentation. Simply allowing any example tangentially related to Rust would probably lead to a poor quality resource, for this to be useful you would want each topic to have a sufficient number of coherent examples exploring the topic that it would be worth checking out rather than using the existing ecosystem documentation.

[mdinger]

I was thinking about this same thing when reading that thread. As one of the main writers for the site (been a while since I wrote anything though), when I write I usually focus on concept type topics such as expanding topics the book covered but trying to do it in a different way while exploiting the different topic style as best I can. This is especially true when I think a topic is quite difficult to understand. Some examples would include generics and error handling. Some extensions I might expect would be an improved macros section, adding the ? to error handling when it lands stable, explanation of impl specialization and such.

While expansion to the documentation, testing, and module structure would be reasonable, I never focused on them because they aren't runnable. Inline runnable examples are pretty much a highlight of the site so focusing them heavily seems reasonable. Also noteworthy, external crate documentation won't be runnable.

The fact that TOC is already extensive would give me pause before attempting to integrate anything other huge "meta documents" into it. Technically, it is already mostly a big concept section followed by std library extras.

This raises another issue: the std library docs which should technically be updated with the std library aren't necessarily being updated. When they stop compiling, we fix them but probably not before. This is related to them being far from the std library source code and out of sight, out of mind. This raises the question: would it be good if the site itself was split into two meta sites, a basic rust concept examples and general std library site?

The questions which could be raised could go on but I'd like to point out an interesting documentation tool for F# in comparison, FSharpFormatting (the page is rendered with the library by the way). This site has an automatic API reference generated and linked to the API Reference linked at the bottom right. The rest of the pages are automatically generated by based on some structure the author provides. These provide a more standard documentation style than the API docs do.

Without question, rustdoc generates better API docs than this library. What FSharpFormatting allows though, is the easy ability for a library to have an all inclusive site documenting their tool without shoehorning the documentation into the API format.

This is not to say rust needs a something exactly like this. What it does say is that, if this tool was the standard tool for rust, we wouldn't need rustbyexample clones. We'd need the library developers to fill out their own examples sections or get community members to add them.

[lambda]

Yes, I've been considering whether the better approach would be a more consistent, unified set of tooling, and a push to get examples included in library documentation using said tools.

Another example of a tool for inspiration is Sphinx, which is widely used in the Python community but also outside of it, and provides tools for integrating API docs with long-form docs. I don't think that just adopting Sphinx would work; the Rust community has pretty firmly settled on Markdown, so it's probably the better approach to work on making Markdown tools that work well with Rust for this purpose, rather than switching to Sphinx/reStructuredText.

While looking into this, I've found that there are a good number of tools out there for various types of Rust documentation, but there isn't one unified toolchain that everyone is using. There's rustdoc for API docs, which is pretty universal. There's rustbook, which is used for the Rust Book and maintained in the rust source tree, but no one really seems to be working on it. There's gitbook, used by Rust by Example currently. There's mdBook, used by the new, work in progress Rust Book, as well as there being an intent to use it for Rust by Example at some point in the future. And then there are little literate programming tools like rust-skeptic and tango.

It seems like it might be nice to try to factor out a set of libraries that a common toolchain for API docs, examples, and long-form book style documentation, and possibly a common output product that would make it easy to have single documentation front page for a project that contains a mix of API docs, examples, and long-form docs.

It seems like there's already a good Markdown backend available, in the form of pulldown-cmark; that's used by at leas mdBook and rust-skeptic, I believe.

There isn't currently a good way, as far as I know, to get API information out of Rust code outside of rustdoc, which seems to be more suited for just compiling straight to a full set of web pages and index information useful only for its search functionality, rather than as a general purpose interface. If some of that functionality could be extracted out of rustdoc to either be used as an API, or compiling into general-purpose consumable JSON or something of the sort, that might help in writing a tool like described.

Beyond that, you'd want some syntax highlighting, though you could work around that for now by using a JS based library, and a way to construct a unified table of contents and searchable index; it can be nice to be able to search both API docs and other docs, without having to go outside through Google.

So, it seems like the first thing that would be needed for making a better unified documentation tool would be to be able to extract more information from Rust source code. Of course, there's the Rust Language Server RFC about providing a general purpose tool for extracting this kind of information for IDEs and other tools, which could include documentation generators. But I don't know if there has been any progress yet on actually implementing this. Another option would be to try rewriting rustdoc to either be usable as a library, or provide a JSON interface, as an incremental step until RLS is available. Or I suppose you could always use syntex_syntax to extract this information, which looks like what racer does. Or you could implement a Rust parser yourself, though that seems like a lot of work, especially to keep in sync with new features coming from upstream.

Anyhow, I have diverged fairly far from the original question. We don't have these tools available currently, so the question is, where is effort better spent; working on tools, or using the tools we have to write some companion books on a wider range of cross-ecosystem docs, and if the latter, is it better to have one centralized project for cross-ecosystem examples, or try to work with individual high-profile projects to start setting an example?

[mdinger]

Either is kinda okay though the tooling system is definitely the better long term goal. The question I have is if the current situation is really so bad as a stop gap. rust has really good docs for its age and maturity. Many libraries aren't even completely formed. The lack of awesome documentation is just an piece of that immaturity.

Besides, we don't need to make wide sweeping decisions about the current course of all rust documentation based on a thread like that. If someone writes 100 pages of awesome documentation for this site, we can decide to merge or not. If we get an extra 500 pages of awesome documentation about lots of stuff, we can divide the site into categories or something.

Basically, if you want to make a rbe for iron and diesel, write them and make them awesome. If it makes sense to merge these awesome sites together for some reason, then that could be done. It's just a lot easier to deal with concrete entities than fictitious entities someone hopes will eventually exist. Let's be practical. When these awesome tools exist, we can deprecate all other sites in favor of them.

[steveklabnik]

The Rust Cookbook now exists for this. 👍