A binary data "streams+" API & implementations via data producers, data consumers, and pull flow.
The name? BLOB ā Matteo Collina.
Bytes Over Buffers ā Thomas Watson
This is a Node.js strategic initiative aiming to improve Node.js streaming data interfaces, both within Node.js core internally, and hopefully also as future public APIs.
The following modules contain usable components (sources, sinks, or transforms) and are published to npm.
The following modules are not published but are 'functional'.
The following files serve as the API's reference:
bob.h
- The C++ header in 'bob-base'The composition of the classes looks like this:
const { Stream } = require('bob-streams')
const source = new Source(/* args */)
const xform = new Transform(/* args */)
const sink = new Sink(/* args */)
const stream = new Stream(source, xform, sink)
stream.start(error => {
// The stream is finished when this is called.
})
An entire passthrough could look like this:
class PassThrough {
bindSource (source) {
source.bindSink(this)
this.source = source
return this
}
bindSink (sink) {
this.sink = sink
}
next (status, error, buffer, bytes) {
this.sink.next(status, error, buffer, bytes)
}
pull (error, buffer) {
this.source.pull(error, buffer)
}
}
The following files serve as API extension references:
High-level timeline:
All of these steps necessitate the buy-in of many stakeholders, both in Node.js core and the greater Node.js ecosystem. This is a long-term project by necessity and design.
Some collective goals for this initiative.
As a preface, "protocol" refers to a system with "producer / source" and "consumer / sink" endpoints.
The Protocol itself must be simple:
Please see performance.md for profiling results & information.
Current results estimate a 30% decrease of CPU time in bad cases, and up to 8x decrease in good cases. This should correlate to overall throughput but may not be exact.
API reference examples sit in the top-level directory and are prefixed by reference-
.
These are functional and tested when practical, notably reference-verify
, reference-passthrough
, and verify-buffered-transform
.
Other helpers, such as Stream()
, reside in the /helpers/
and /tests/helpers
directories.
All useful and usable components in this repo are exported from index.js
with the bob-streams
npm module.
Functional sources, sinks, and so on can be found in their own npm modules. See [Published Modules](#Published Modules).
npm install && npm test
The addons are presently very out-of-date.
You must have a local install of Node master @ ~ 694ac6de5ba2591c8d3d56017b2423bd3e39f769
npm i node-gyp
node-gyp rebuild --nodedir=your/local/node/dir -C ./addons/passthrough
node-gyp rebuild --nodedir=your/local/node/dir -C ./addons/fs-sink
node-gyp rebuild --nodedir=your/local/node/dir -C ./addons/fs-source