Passing essential meta data between streams (Stylus example)

[owenb]

Hey!

This 'forum' is a great idea.

I've written a basic wrapper around Stylus so I can pipe Stylus code in and get CSS out:

https://github.com/socketstream/socketstream-0.4/blob/master/example_app/stylus-stream.js

Problem is, Stylus needs to know the file's directory to allow you to @include another file.

Stream Team: What is the best way to send this meta data?

[isaacs]

What about something like this?

var input = fs.createReadStream('file.styl')
var output = fs.createWriteStream('file.css')
var thingie = stylus.createStream({ directory: process.cwd() })
input.pipe(thingie).pipe(output)

That is, just an arg when you create the transform stream.

[Raynos]

:+1: to stream factories!

[dominictarr]

doesn't @mikeal do something like this with filed?

you can get a handle on the previous stream by setting a listener on 'pipe'

var path = require('path')
dest.on('pipe', function (source) {
  path.dirname(source.path) //will only work if this is really a fs.ReadStream
})

we've been talking about having metadata of headers on streams so that you can do clever stuff like this automatically, but I seems potentially complicated/messy. I have some other applications where this would be useful too.

@mikeal does this sort of stuff in request and filed, so those might be worth a read.

[owenb]

Thanks all. Still playing around with this idea at the moment. Trying to avoid creating a brand new stream on each incoming HTTP requests for performance reasons, so I'll try the dest.on('pipe') approach and see if that works.

[dominictarr]

@owenb what do you mean "trying to avoid creating a brand new Stream on each request" are you talking about object pools?

The slowest thing by far is the IO, if you are writing a stream across a network (especially, across a mobile network) it's highly unlikely that the app becomes CPU bound (that is, for an extra stream, a few more functions, to matter)

[mikeal]

it's super easy, just add a listener for the "pipe" event in the constructor for your stream.

[brianloveswords]

ok so heres a contrived example of the request magic pipe stuff, right?

// in a client
var file = fs.createReadStream('file.css')
var request1 = request({'url':'myproxy.com', headers: {'x-foo': 'bar'}})
file.pipe(request1)

// in a server
function requestHandler(req, resp) {
  var request2 = request({'url': 'google.com'})
  req.pipe(request2).pipe(resp)
}

when req pipes to request2 request knows they are both request objects because of the on('pipe') listener and it copies all headers and other http metadata crap to request2. the nice part about this is since it is transmitting the data over http (which has metadata built in via headers) the metadata persists over network transmission

IS THIS ACCURATE/ OOPS SORRY CAPS LOCK

[brianloveswords]

another example: given a stylus stream like this:

// in a client
var stylusStream = stylus.createReadStream({directory: process.cwd})
stylusStream.pipe(request('http://awesomewebsite.com'))

// in a server
function requestHandler(req, resp) {
  // req only has a stylus spreadsheet body in it... 
  // the directory would have been lost when it got piped to an http request
}

so why can't we just use http semantics for every stream in terms of metadata so that if a stream with metadata gets piped over the network we can have the metadata transported as http headers?

[dominictarr]

hmm, so you could do something like this, and then also depending on the stream, the headers could also be written out to the data - of course, this would depend on the type of stream.

Largely, this would be a matter of having transform streams just copy headers/metadata across, even if they don't care about them. If there someone figured out a good scheme for this, I'd be willing to add this as a feature to through, which means it would apply to these 46 depending streams too https://npmjs.org/browse/depended/through

A stream could also serialize the headers, and allow it to travel over a text stream. This could be pretty cool for attaching metadata to files, etc, people would publish all the modules which defined stream types headers to npm. Then you could automatically detect what type a stream/file is.

This idea still feels a little hairy to me, but, it's probably a good idea.

[max-mapper]

here is a generalized pattern:

var streamInstance = new StreamPrototype(metadataObject)

wouldn't it be awesome if request knew to take metadataObject when pipe happens and transport it as x-headers in the http-request or something?