search

sindresorhus / get-stream

Get a stream as a string, Buffer, ArrayBuffer or array
MIT License
341 stars 33 forks source link

Add a `maxBuffer` option #8

Closed jamestalmage closed 8 years ago

jamestalmage commented 8 years ago

See https://github.com/sindresorhus/execa/issues/33

sindresorhus commented 8 years ago

👍 I assume you want an error thrown when it reaches the maximum. For the array method, maybe it should count array items instead of bytes?

jamestalmage commented 8 years ago

My thought was that if you set maxBuffer, every item passed in must have a numeric length property.

Maybe a maxItems option, only for arrays? That way you can still use maxBuffer for arrays if there is a length property

sindresorhus commented 8 years ago

Ok, so how about this; We do the same for all the methods and limit on the total byte size, but if it's an object stream (accepted in .array()), we do the same as the stream highWaterMark option and work on items, not on byte size. ?

sindresorhus commented 8 years ago

When in objectMode, streams can push Strings and Buffers as well as any other JavaScript object. Another major difference is that when in objectMode, the internal buffering algorithm counts objects rather than bytes. This means if we have a Transform stream with the highWaterMark option set to 5, the stream will only buffer a maximum of 5 objects internally. - https://nodesource.com/blog/understanding-object-streams/

sindresorhus commented 8 years ago

Sidenote: Object streams feels very similar to Observables, although with back-pressure support and no utility methods.

jamestalmage commented 8 years ago

Re: Observables

Yep, that's it essentially.

jamestalmage commented 8 years ago

A stream of strings is still in objectMode though (I believe). So you couldn't do:

getStream(stream.pipe (split()))
sindresorhus commented 8 years ago

Are you sure? I don't see any objectMode reference here: https://github.com/dominictarr/split/blob/master/index.js

jamestalmage commented 8 years ago

No. Definitely not sure.

jamestalmage commented 8 years ago

From the article:

If an object stream isn't emitting Strings or Buffers, it's important to note that you can't pipe it to a non-object stream.

So it looks like maybe objectMode is taken into account on the stream you are piping to?

So if I'm setting up a transform stream that takes in Strings/Buffers and emits Objects, is that not an ObjectMode stream (because it doesn't take objects)? I think that's what I'm reading.

kevva commented 8 years ago

So if I'm setting up a transform stream that takes in Strings/Buffers and emits Objects, is that not an ObjectMode stream (because it doesn't take objects)? I think that's what I'm reading.

Yup, I think only the receiving one needs to be.

jamestalmage commented 8 years ago

https://github.com/jamestalmage2/stream-objectmode-tests

That turned out to have a few surprises.

sindresorhus commented 8 years ago

@jamestalmage That's useful info! You should submit a link to the repo to https://github.com/stephenplusplus/stream-faqs ;)