Visualize the internal state machines

[domenic]

This would be an immensely helpful illustrative diagram to make.

[domenic]

Sketching this out also revealed a few questions. I think this is the correct way to drive development and will be using the below to rewrite the spec's state for clarity and robustness. Please look this over.

Operations

• [[push]], [[finish]], [[error]]: called by start or pull constructor parameters.
• read(), abort(), waitForReadable() external operations.

States

• waiting
• readable
• errored
• finished

Breakdown of States x Operations

Waiting

• [[push]]: moves to readable
• [[error]]: moves to errored
• [[finish]]: moves to finished
• read(): no state change
  • throws an error side effect (or could return BUFFER_EMPTY signal? #26)
• abort(): moves to finished
  • calls [[abort]] side effect
• waitForReadable(): no state change
  • calls pull side effect (see #26)

Readable

• [[push]]: no state change
  • pushes on to buffer side effect
• [[error]]: moves to errored (see discussion in #25)
  • clears buffer side effect
• [[finish]]: no state change
  • sets finished flag side effect
• read():
  • shifts off of buffer side effect
  • if buffer is empty and finished flag is set: moves to finished
  • if buffer is empty and finished flag is not set: moves to waiting
  • if buffer is not empty: no state change
• abort(): moves to finished
  • clears buffer side effect
  • calls [[abort]] side effect
• waitForReadable(): no state change

Errored

• [[push]]: no state change
  • Ignored (or could throw?)
• [[error]]: no state change
  • Ignored (or could throw?)
• [[finish]]: no state change
  • Ignored (or could throw?)
• read(): no state change
  • throws error side effect
• abort(): no state change
  • Ignored? Or could throw/reject? Or could trigger [[abort]] anyway?
• waitForReadable(): no state change

Finished

• [[push]]: no state change
  • Ignored (or could throw?)
• [[error]]: no state change
  • Ignored (or could throw?)
• [[finish]]: no state change
  • Ignored (or could throw?)
• read(): no state change
  • throws already finished error side effect (or could return EOF signal?)
• abort(): no state change
  • Ignored? Or could throw/reject?
• waitForReadable(): no state change

[creationix]

First off, I really like this style of spec. I write state machines all the time and it's really natural for me. Also it forces you to see all the edge cases formally.

[creationix]

• in "Errored" and "Finished" states, I think abort() should be ignored. The consumer doesn't care what the producer does with it.

I'm not quite clear on what waitForReadable() actually does. I assume it will return a promise or take a callback to notify the caller when the stream is readable.

[domenic]

I'm not quite clear on what waitForReadable() actually does. I assume it will return a promise or take a callback to notify the caller when the stream is readable.

Right, I left out what it actually does. You're right that's what it's supposed to do. I need to add side effects internally to reset, resolve, or reject the internal promise it returns at the appropriate steps.

Another point: I need to prevent calling pull more than once. This may need another state ("reading?"), or perhaps just a flag.

[SimonSapin]

I’ll just leave this here. https://github.com/tabatkins/railroad-diagrams/

[mattpodwysocki]

I like this way of visualizing through what we call marble diagrams in Rx: http://netflix.github.io/RxJava/javadoc/

[benjchristensen]

Deep link to a page with the marble diagrams: http://netflix.github.io/RxJava/javadoc/rx/Observable.html and one of the wiki pages with them: https://github.com/Netflix/RxJava/wiki/Combining-Observables

I very much want dynamic "Chrome/Webkit Profiler" style visualizations (with relationship between streams added) for these types of systems. That's something we are exploring for tracing live systems.

[trevnorris]

I see a few implementation questions that may or may not need to be addressed here, but I'll bring them up.

• IMO [[readable]] should only fire once after data has been received when the stream is in a drained state. Other than that the user knows data is ready to be read, and it's up to them to do something about it.
• If the stream is to continually receive data, and since this is following more a pull based model, some sort of automated back pressure mechanism needs to be in place.
• Also with this in mind the user should be alerted to a point when it's impossible any additional data can be read().
• Operations like read() should accept a callback. Say for two reasons. One is so any errors can be passed to the callback, instead of having the call throw (IMO allowing multiple return types like Buffer/ArrayBuffer and Error will cause more issues than it solves). Also it gives more options that can be passed in. Say like to number of bytes remaining to be read. Or a sort of highWatermark approach like Node uses were a user passes in N and the callback doesn't get called until that amount of data is available.
• Maybe [[finish]] addresses this, but need a way to prevent more data from being written but also allow remaining data to finish being read from the receiving end.

[domenic]

@trevnorris thanks for your thoughts; really great to have you join the conversation.

• Yeah, stream.waitForReadable().then(onReadable) will only ever call onReadable once, when the stream is the drained state.
• Automatic backpressure is a key feature of streams, no doubt! What is missing right now is details on the high water mark strategy; right now, push always returns false, i.e. the high water mark is effectively 0. But easy high water marks are planned, with some default easy-to-install strategies like "length counter" (for string and ArrayBuffer streams), or "counter" (for object streams), or customizable ones.
• Yeah, the user can do stream.finished.then(noMoreDataBecauseSuccess, noMoreDataBecauseError)
• I don't think callbacks or N bytes fits with the synchronous read model we want to provide, at least on the primitive level. But it should be easy to build such helpers, e.g. the getNext example for getting the next element with a callback, or the readableStreamToArray example for getting all elements; clearly you could create a hybrid that waits for N elements or N bytes before fulfilling the returned promise. User space is probably the place for building such helpers; our goal with the synchronous read interface is to provide the most performant and low-level primitive.
• Yeah, exactly, the intent of [[finish]] is not to immediately transition into the finished state, but to set a finished flag so that no more data can be written, and once the buffer is trained you transition into the finished state.

[domenic]

I rewrote things to use a state machine. I think it might be possible to remove the "pulling," "draining," and "opening" flags in favor of more internal states (i.e. ones not exposed through readableState). But this at least fixes a lot of issues and edge cases. Time to move on to writable stream and the toolbox.

[domenic]

Here is a draft of the state machine for readable stream:

http://www.gliffy.com/go/publish/5148129

Feedback welcome. Maybe todo:

• loopback .read()s on readable states. (Not sure how to phrase this; ".read(), buffer empty" goes to waiting/finished, vs. ".read(), buffer not yet empty" loops back to readable?)
• Should probably be vertical not horizontal
• Make it prettier

[aredridel]

++

[domenic]

Two writable stream ideas, one with loopbacks:

http://www.gliffy.com/go/publish/6388889

one without:

http://www.gliffy.com/go/publish/6388970

Could add loopbacks to readable stream too, unsure.

Would be interested in review before SVG-izing this.

[tyoshino]

The initial state is "Waiting" in the diagram but is "writable" in the spec. Let's discuss in #242