book: proposal for introduction and overall theme

[nrc]

I've been pondering how best to present the book. My feeling is that the approach that most tutorials take of going bottom up from event loops and how futures are scheduled up to the high level concepts like async/await is not great for beginners to the area - there are a lot of moving parts to get your head around, lots of jargon, and lots of libraries, most of which take some hand-waving by the author.

I believe a better way is to start with the concept of async code and async/await. Then move on to futures in the abstract and then drill into the details of event loops and how it all works. Then the reader can concentrate on the mental model of async at first, before having to also learn the details.

I started hashing out a rough outline, but I quickly realised I didn't know the area well enough. Instead I wrote just the introduction (taking some pieces from the old apr book). It relies heavily on examples which use async/await and minimal external libraries (the final example does use bits of tokio in order to get something interactive, but hopefully it is not overwhelming in the way that Hyper examples can be).

The intro draft is at https://github.com/nrc/apr-intro. If others like the sound of this approach, I can polish and submit a PR. I'm happy to work on more too, but my time is a little scarce at the moment.

cc @cramertj @aturon @mgattozzi

[yoshuawuyts]

@nrc I love this! Especially the flow of how the concepts are introduced felt really good!

[cramertj]

@nrc We spent a while talking this over this last weekend and at RustConf (and in numerous previous discussions) and the general consensus has been that the ecosystem is not yet at a point where developers can be productive only understanding async/await at a surface-syntax level.

We wrote up a TOC (i'll link it here when I track it down, cc @aturon) which starts off by introducing users to async code via async/await, and then follows that up with a "what's going on here?" which goes into some more of the details and examples we covered in the async class.

After that, the book would return to async/await style and cover common patterns and "how do I do X"/"fix X compiler error"-style explanations of different features, including some of the most common combinators and the macros like select! and join!. I'll make sure to paste the TOC here when I find it and we can discuss further.

[tinaun]

https://paper.dropbox.com/doc/APR-RustConf-notes-ILUjs8lJjFgqiAuduz0mN

[nrc]

the general consensus has been that the ecosystem is not yet at a point ...

Tokio async was released this week, imo, that changes the calculus here.

can be productive only understanding async/await at a surface-syntax level

this is probably true, however, I think they can productively learn about async programming this way, at least the fundamentals. This intro is meant to be a proof of concept of that approach.

which starts off by introducing users to async code via async/await

hopefully this intro fits that bill?

and then follows that up with a "what's going on here?"

IMO, this is getting into the nuts and bolts too early. There are a huge number of moving parts here and they should be introduced one at a time. In particular, after async/await, I think you can introduce futures without talking at all about executors, waiters, and so forth.

After that, the book would return to async/await style

Not 100% sure what this entails, but I think, e.g., async fn vs async {} and similar details could be handled before even introducing futures. In particular, I think it is essential to have a good understanding of the async model so that the details (even futures) can be understood in that context (there are plenty of subtle but important differences between our model and others). I think digging into some 'advanced' async/await stuff is a good opportunity to ensure the reader really gets the mental model right.

cover common patterns and "how do I do X"/"fix X compiler error"-style explanations of different features...

at this point I don't know the ecosystem/features well enough to have an opinion, but that sounds good to me.

https://paper.dropbox.com/doc/APR-RustConf-notes-ILUjs8lJjFgqiAuduz0mN

I'll leave a few comments there

[tinaun]

tokio-async-await is still based around the futures 0.1 core trait defintions, which might complicate things - i haven't played with it enough to see how it deals with converting async blocks into 0.1 compatable futures, though.

[Nemo157]

@tinaun it doesn’t in general, it only allows you to spawn a 0.3 future as a task, the shim to convert 0.3 -> 0.1 is a hidden implementation detail. Similarly the shim to go 0.1 -> 0.3 is a hidden implementation detail of the await! override.

[cramertj]

@nrc and others planning on helping w/ the book (I think @tinaun, @Nemo157, @mgattozzi): my plan was to start by putting together the first couple of chapters throughout the course of next week, initially following the chapter layout we discussed at RustConf (linked in the paper above). Once that is reviewed and we've gotten an idea what we want the overall look / tone of the book to be, we can break out the rest of into subsections that we can each tackle. If we decide that the book doesn't need to include the low-level introduction, that material will still be worthwhile and we can break it out into a separate document, so my plan will be to go ahead and write it, and we can do a pass afterwards and decide if the material makes sense to include in the book or not.

How does that plan sound to you all? Are there changes you'd like to make to this plan? Are there specific sections any of you really want to write?

[nrc]

If we decide that the book doesn't need to include the low-level introduction, that material will still be worthwhile and we can break it out into a separate document

This seems (potentially) pretty wasteful unless you've already started writing - I imagine that even if we're covering the same material, we'd do it very differently if its the introduction or the nth chapter.

initially following the chapter layout we discussed at RustConf

I think it is worth reconsidering this. The major change since Rustconf is that we can use enough of the ecosystem to write a satisfactory introduction.

It might be worth being explicit about the audience. The audience I have in mind is intermediate or advanced Rust programmers with possibly no experience with async or network programming. An alternate audience might be programmers with only a little Rust experience, but with some async/network programming experience.

Certainly in the former case, I think it is a mistake to work bottom up, now that we can go top down. Top down means introducing concepts one at a time and always keeping the goal in mind. Bottom up it is hard to be motivating and effectively introduces everything (details and abstract concepts) at once. For someone not deeply involved with this stuff, it is really hard to make the connection between state machines and executors and wakers, etc. and the actual goal of async programming.

Once that is reviewed and we've gotten an idea what we want the overall look / tone of the book to be, we can break out the rest of into subsections that we can each tackle

I don't have much time myself, but if others do, then it would make sense to plan now and divide up the work early rather than gating on writing the first couple of chapters. Is there a reason you want to finish those chapters before starting on the rest?

Are there specific sections any of you really want to write?

I don't have a particular preference, happy to help out where it is most useful. I think I can probably have a better impact on the introductory material than the advanced stuff since I'm probably the least expert one here.

[aturon]

@nrc To fill in some missing context: the Tokio compatibility layer was already in the works when the current table of contents was worked out, and was taken into account. After RustConf, several of us worked through a very similar discussion about the feasibility of various approaches to the material. The table of contents here is not purely top-down or bottom-up, but rather a mixture aimed to get people productive quickly, but to teach a couple of the fundamental parts of async in Rust that tend to trip people up -- namely the task model -- which will continue to be an issue even as we move increasingly toward async/await.

I think part of the disconnect here is that the outline, as written, doesn't give the full context from the discussion on what would actually go in to these various sections. So e.g. the section on futures early on is not intended as a comprehensive view or to spend time on combinators etc, but instead to give you the bare minimum understanding needed to work effectively with async code in Rust. By far the emphasis is on working with async/await.

Is there a reason you want to finish those chapters before starting on the rest?

Cohesion -- setting the style, tone, and foundation for the book early will make it much easier for the other chapters to fit into a cohesive whole. Note also that this book is @cramertj's top priority at work, where he already does a lot of async/await/futures training, so I expect things will move quickly!

[nrc]

Thanks for the extra context.

but rather a mixture aimed to get people productive quickly, but to teach a couple of the fundamental parts of async in Rust that tend to trip people up

I think my (controversial) opinion is that these two goals are mutually exclusive, at least in the same chapter/section. Obviously any book would want to both and more. But I'll hold back and wait to see a more fleshed out plan/draft of the early chapters.

I think part of the disconnect here is that the outline, as written, doesn't give the full context from the discussion on what would actually go in to these various sections.

It would be great to flesh this out over the next week or so as @cramertj is working on the foundation of the book. I think it will make it much easier to contribute.

[cramertj]

@nrc

I think my (controversial) opinion is that these two goals are mutually exclusive, at least in the same chapter/section.

Actually, I totally agree! I think that it's hard to mix-and-match these, as they require dropping down to different levels of abstraction, and I think the book should tackle them in largely separate sections. However, I think it's necessary (even with tokio!) that developers understand both mechanisms, as developers often have to drop down to lower levels, especially when debugging. I think the book should cover both, but that there should be a clear distinction, and that users who are interested in getting the highest-level view of async/await should be able to do so.

It would be great to flesh this out over the next week or so as @cramertj is working on the foundation of the book. I think it will make it much easier to contribute.

Yeah absolutely! I'm partway through a draft of a couple chapters now-- I'll try to have the first few and outlines of the rest posted this week so that we can have more concrete materials to chat about. I think it'd help to have something tangible we can discuss, even if it's not what we decide on for the final structure.

[steveklabnik]

I'd like to mention that the reference is growing an introduction similar to TRPL:

• https://doc.rust-lang.org/book/2018-edition/ch00-00-introduction.html

• https://doc.rust-lang.org/nightly/reference/

• What The Reference is Not

• How to Use This Book

• Conventions

• Contributing

I think something like this would be a good thing for any introduction to any of the WG books. Tweaked for each situation, of course, but the bottom three are generally quite useful no matter the topic.