Streams cookbook

[mafintosh]

Bunch of examples for writing streams.

A series of examples on how to write different kind of pipelines etc.

[indexzero]

Hot take on TOC for this:

• Using PassThrough, through2, and "pipe" events
• Error handling, invoking callback and artificial back pressure. (see: https://github.com/winstonjs/winston/pull/1347)
• Implementing objectMode streams.

[mafintosh]

Thoughts:

• You prob always want to use readable-stream if you are writing a module

Cookbook example. A stream that writes first to a tmp file, then to a normal file.

const stream = require('readable-stream')

const fd = fs.openSync('file.tmp', 'w')
const ws = stream.Writable({
  write: function (data, enc, cb) {
    write(fd, data, cb)
  },
  // using the final method we can make sure the 'finish' event
  // happens *after* the file has been renamed, which makes composition
  // a lot easier.
  final: function (cb) {
    fs.close(fd, function (err) {
      if (err) return cb(err)
      fs.rename('file.tmp', 'file', cb)
    })
  }
})

// a simple function that writes all data to a file
function write (fd, data, cb) {
  fs.write(fd, data, 0, data.length, function (err, wrote) {
    if (err) return cb(err)
    data = data.slice(wrote)
    if (data.length) write(fd, data, cb)
    else cb()
  })
}

I think the cookbook should just be a bunch of those

[mafintosh]

Another example:

const duplexify = require('duplexify')
const http = require('http')

// duplex http stream
function request (opts) {
  const stream = duplexify()
  const req = http.request(opts, function (res) {
    stream.setReadable(res)
  })
  stream.setWritable(req)
  return stream
}

[mcollina]

cc @amiller-gh - this is the guide we have been talking about.

[amiller-gh]

👍 @add1sun has kicked off the documentation location/format discussion here: https://github.com/nodejs/community-committee/issues/329

Like we talked about at the conf, we don't have any technical plans laid out yet for how to store, ingest, and display docs that live alongside – so this is the perfect time to kick off the conversation!

This shouldn't stop you from writing content though. Feel free to start creating more detailed guides for streams in this repo and we can re-format as needed once the website architecture is fleshed out. If we go the in-repo, multiple-source route, it would be great to have readable-stream as a guinea pig for the new ingestion pipeline 🙂