Streams Planning

[mikeal]

I'd like to use this ticket for planning the streams session. I'll go ahead and paste in what I sent you guys in email.

On the survey, streams came up a lot as something people wanted to learn more about but hadn't had the time.

I would suggest this: write a 5 minute talk about using streams in node and why they are awesome, then assign everyone a task they should be able to complete in 20 minutes. Then do another 5 minute talk about streams in the browser, and assign two optional tasks, one for the people who really don't understand the browser that is strictly node and another that uses voxels (if you think you can do a voxel intro in a short enough amount of time).

The 8 sessions take us a day and a half. The second half of the last day is more freeform hacktime and I'm thinking of combing sessions. So the "streams" room would all of a sudden be "streams and hardware" or "streams and copters" or "streams and distributed datastructures" and you'd have all the staff from both session and people could tackle more advanced topics. We can combine which ever session you think complements voxels better :)

@maxogden @substack

[ghost]

Here are some ideas for stream activities:

• implement a streaming interface using raw streams from core
• implement a streaming interface using through
• backpressure and streams2
• meta-streams like mux-demux
• streaming parsers
• protocol streams like dnode and scuttlebutt

To make this session more fun and voxely some of these challenges could take place in a voxel environment with actual 3d plumbing that uses node-style streaming interfaces.

[dominictarr]

Incidentally, I'm gonna give a workshop along these lines at campjs.

[mikeal]

this needs to be a mix of actually teaching people something and then having them implement was taught as well as room for them to go off and do crazy stuff.

some things to keep in mind.

• the majority of the attendees barely know what a stream is or how it works
• the vast majority of attendees will not have written a stream before
• not all attendees do client side browser programming
• because there is only an hour, and 40 people, attendees need to be able to answer some of each others questions, we can't rely on the 3 or 4 people running the session to be holding everyone's hand at once. this makes the cost of task diversity very high.

i'm going to create another issue for a discussion on "choose your own adventure" methodology.

[domenic]

I'd also like to add:

• the vast majority of attendees will be confused and dazed by streams2 having recently replaced anything they might have vaguely known before

[mikeal]

per @domenic's point, depending on the adoption of 0.10 we either only teach streams1 or only teach streams2. a discussion of both will take the entire hour.

[hueniverse]

We should just team streams2. This is what we need people to know. 0.10 will be out and usable, regardless if they are ready to upgrade to it or not.

[max-mapper]

this is a pretty good article, maybe we can send it to attendees ahead of time http://loose-bits.com/2012/08/02/nodejs-read-write-streams-pipes.html

[Raynos]

streams2 are bloody hard to get right. There are way too many edge cases and complexities.

I would recommend that if you consider teaching streams2 you deep dive into them. You'll probably change your mind.

[ghost]

I haven't even written any streams2 code. Teaching a session on it wouldn't make sense. There's too much uncertainty about streams2 right now to be teaching this stuff to beginners anyways.

[mikeal]

June is a ways away :) if we've all moved to streams2 we'll teach streams2. if not, we'll stick with streams1. knowing that they have a lot of added complexity i'd be more inclined to use APIs that make them simpler.

[max-mapper]

the point isnt to teach people the boring specifics of implementing a stream but more how to use streams in their apps as opposed to not using them at all (which is how the majority of people operate now)

[Raynos]

That's a good point actually. Implementing a stream is what you do for a small subset. The interesting part is using many streaming apis and demonstrating how easily they inter operate together with pipe directly or with some intermediate through / transform streams

[dominictarr]

Exactly, write end "data" etc, should be considered private methods. pipe should be considered to be the only public method, if possible.

[domenic]

I was just reminded of how much people need a very basic course. None of the fancy meta or protocol or bidi or gossip streams. No JSON streams, no parsing streams, no browserify streams. Just, how to use streams in HTTP applications for data.

My reminder came in the form of being brought onto a project where coworkers are using an Express middleware to convert incoming image requests into a buffer before saving it, because this is apparently easier. (headdesk)

I think at least the first segment, and maybe the entire session, should be focused on just using streams everywhere, and pounding into attendees heads that you should never buffer in memory if you can avoid it.

In this vein, useful exercises would be streaming things from a file upload control to the filesystem (or to S3, or to each others' computers, or whatever!) with the emphasis being on streaming files, and never buffering them.

Hope this is a helpful perspective :).

[hueniverse]

My biggest issue with express is that it trains people to think that req.body is part of node.

[mikeal]

maybe the talk should just begin with "req.body is not a thing, stop using it" :)

[domenic]

I honestly think a session centered around "use cases for req.body and what you should be doing instead" would be a big hit.

[mikeal]

remember that there is a broader range of people attending than this. not everyone will be interested in highly scalable streaming servers.

the reason I wanted @substack and @maxogden to handle this session is that they are the closest people to interesting userland streams and end-to-end streams to the browser.

for each person who is actually a fan of express attending there will be someone who just uses express as the quickest way to get something up on the backend so they can do something great on the frontend. to get those people to care about streams we need something interesting that demonstrates their value end-to-end, not just a lecture on buffering.

[hueniverse]

Show a cpu load chart of express with default body parser used as a proxy vs. streams... The streams version just sits there.

[dominictarr]

I did a streams workshop recently at campjs,

Basically, I followed the structure substack used in his lxjs talk harnessing the power of streams

But making it more interactive, and having the participants run the examples themselves, but the same 'plot' if you will,

It covers the gamut of stream use, and each example progressively introduces a new stream concept. The first example parsing a large (100's MB) json file with JSONStream - first try loading it and parsing in one go, and it takes ages (in my demo I tried to parse all the json in npm, and node crashed), then stream-parse it and not only does it not crash, but it starts producing output immediately!

Following that you write a stream that applies some simple transformation, like capitalising all input...

And then an example where you loose data if you don't buffer/pause an incoming stream, etc...

Anyway, my point is non-http-streams need not be considered 'advanced', I'm sure whatever substack and max do will imply all the things you could do with streams in express.

[hueniverse]

The problem is not express, but the default setup everyone using express is cut/pasting. There is nothing inherently wrong with express or anything to prevent you from doing the right thing. Using steams is more work for most simple forms and small json payloads with is the majority of apps using express.

[mikeal]

this is the Streams session, not the anti-express session :)

[dominictarr]

@hueniverse absolutely. streams are for when you can start acting on something before you have received the entire resource. json/forms with a fixed numer of items, etc, or templates that arn't a massive list, don't benefit from streams.

The cool thing about streams isn't using less memory, but doing the ordinary web app things, but being about to do new things without worrying (too much) about memory.

[max-mapper]

for inspiration on topics to teach: http://imgur.com/a/9vFGa#0

On Tue, Feb 26, 2013 at 5:39 PM, Dominic Tarr notifications@github.comwrote:

@hueniverse https://github.com/hueniverse absolutely. streams are for when you can start acting on something before you have received the entire resource. json/forms with a fixed numer of items, etc, or templates that arn't a massive list, don't benefit from streams.

The cool thing about streams isn't using less memory, but doing the ordinary web app things, but being about to do new things without worrying (too much) about memory.

— Reply to this email directly or view it on GitHubhttps://github.com/mikeal/nodeconf2013/issues/2#issuecomment-14151195 .

[mikeal]

adding @mattpodwysocki to this session.

[ghost]

I don't like streams2. I won't be talking about it, but it isn't important anyways. People shouldn't be writing streaming code with the raw apis. I'll talk about through, duplexer, concat-stream and similar libraries instead and how you can pipe them all together to get work done. Implementation details like streams1 vs streams2 are really boring. That we have a thing called "streams2" at all just confused people but they should just ignore it completely and pretend that it doesn't exist.

[hueniverse]

@substack I disagree. People need to understand how the basic core API works and how they can easily write their own stream objects. We also need to move people away from the data/end events and teaching about the new readable event is really important.

[ghost]

Why even bother writing your own raw stream when you can just do:

var through = require('through');
module.exports = function () {
    return through(function (buf) {
        this.queue(String(buf).toUpperCase());
    });
};

and it doesn't even matter if you're using streams1 or streams2 because it shouldn't matter.

[Raynos]

Why even bother using modules when you can do:

var Transform = require("stream").Transform;
module.exports = function () {
  var s = new Transform();
  s._transform = function (buf, _, cb) {
    this.push(String(buf).toUpperCase());
    cb();
  };
  return s;
};

[Raynos]

We also need to move people away from the data/end events and teaching about the new readable event is really important.

I recommend you teach people that readable.pipe(writable) always works and that "data" or "readable" are implementation details.

In most cases you should create a custom stream and pipe into it because that handles a lot of edge cases. Either using require("through") or require("stream").Writable

[mikeal]

@substack I know you want to have people hook together a lot of streams using common modules. Could you please install any modules you need in to this repository and check them in. We're assuming the wireless will crash during the conference and will be sending out this repo as a tarball and we'll have backups on usb keys during the conference.

[ghost]

All the materials are going here, all in the open.

https://github.com/substack/stream-adventure

This repo is a bad place to iterate on materials.

[mikeal]

You're missing the point @substack, we need to get people to install the package before the conference because the wifi is likely to go down, that's why I'm trying to contain it all in one package, so that people can npm install nodeconf2013.

If you want to write them in another package that nodeconf2013 depends on, that's fine, but you need to finish up by EOD tomorrow because I'm sending out an email on Wednesday to get everyone to install it.

On Jun 17, 2013, at 2:22AM, James Halliday notifications@github.com wrote:

All the materials are going here, all in the open.

https://github.com/substack/stream-adventure

This repo is a bad place to iterate on materials.

— Reply to this email directly or view it on GitHub.

[mikeal]

also, the only thing you need to get in the nodeconf2013 package are the dependencies people need and any materials you want them not to write themselves. If you're going to show some code that you want them to type that doesn't need to be in the nodeconf2013 package.

[mikeal]

ok, i see you already depended on streams-adventure in nodeconf2013, sweet!

[mikeal]

@mattpodwysocki isn't going to make it this year :(