Syntactic assist?

[dead-claudia]

Edit: Changed disambiguator to stream, removed sync variant, clarified. Edit: Added arrow function variant, parallel keyword Edit: Most of this is out of date. Please see this gist.

I was looking at @jhusain's 2-year-old proposal of "push generators" (a glorified Observable), and I was thinking observables might be easier to consume with a syntax assist. Basically, a way of iterating eager sources, much like how generators use for ... of and for await ... of.

Maybe something like this?

// Sync generators
function *foo() {
    await action();
    yield bar;
    yield* gen();
}

// Async generators
async function *foo() {
    await action();
    yield bar;
    yield* gen();
}

// Stream (async context)
stream function foo() {
    await action();
    emit bar;
    emit* obs();
}

// Always available
for (let i of foo()) { ... }

// Only available in stream contexts
for (let i from foo()) { ... }
parallel {
    // Only these statement types allowed
    for (let i from foo()) { ... }
    emit* bar()
}

// Only available in async contexts
for await (let i of foo()) { ... }
for await (let i from foo()) { ... }
await parallel {
    // Only these statement types allowed
    for (let i from foo()) { ... }
    emit* bar()
}

---

As for my specific choice:

• "Stream function": Streams emit values, much like how generators yield values. Also the relation is similar: streams are a common type of observable, like how generators are a common type of iterable, but neither are 100% synonymous.
• emit statement instead of yield: different semantics (push, not pull), and emit may be firing multiple listeners.
• emit* is an expression, because of the potential completion value.
• for ... from: It's iterating events received from the stream/observable. I'm breaking from the for ... on idea, because you're not iterating values on anything. Note that it's only available in stream contexts.
• for await ... from: Sometimes, it's convenient to await for completion. Also, such a promise would be purely an implementation detail, and an implementation might avoid creating the promise at all.
• parallel { ... }: Merging is a very common operation, one that needs first class support. Iteration is inherently sequential, streaming is not. And merging is far more common than concatenation in the world of reactive programming. The word choice is to also open the door for expanding to promises.

And some other notes:

• return and throw work as expected in stream functions, and map to error and complete. (IMHO, the latter are probably not the best names.)
• The body of for ... from and for await ... from is executed in parallel and unbuffered, for performance reasons. Execution continues on the current tick and next tick, respectively, after the observable completes and all outstanding iterations are completed. If you await within that loop or read from another observable, there is the potential for race conditions.
• If either for ... from variant is broken early (via return or break), unsubscribe is called implicitly to avoid memory leaks.
• Stream functions would compile down to async generator-like state machines, but would differ in that they react synchronously to observables.
• parallel { ... } and await parallel { ... } are both expressions, returning an array of their completion values. An implementation might avoid storing them altogether if the result is unused.
• for ... from and for await ... from would require a separate helper.
• parallel { ... } and await parallel { ... } would require a separate helper.
• Stream arrow functions are similar to async arrow functions (just stream instead of async), and are also in an async context.

[appsforartists]

Interesting idea. I'm a bit hesitant to add syntax to the language without a compelling reason - it's easy for it to turn into cruft, and we have a limited number of special characters. If we use @ here, it's unavailable for future use.

Also, it's unclear how you'd complete a stream from within the body of an @function in this proposal. Have you considered how that would work?

[dead-claudia]

@appsforartists

Edit: Update a few things

I'm a bit hesitant to add syntax to the language without a compelling reason - it's easy for it to turn into cruft, and we have a limited number of special characters.

I generally agree with that sentiment myself, and have countered numerous proposals on es-discuss because of it. And in particular, syntactic additions are usually pretty hard to justify. In this case, I feel it's justified because of the following:

1. It makes observation first-class within the language, in the same way generators and for ... of reified iteration. JavaScript is already exceptionally event-driven, so reifying that into syntax would make it easier to work with.
2. It reduces the need for all these operators nearly every reactive library needs, similarly to how generators and for ... of reduced the need for things like Lodash and Lazy.js.
3. It's way more modular and isolated. You no longer have to rely nearly as much on methods for useful functions, and you don't need nearly as many functions for control flow abstraction, much like how async functions reduced the need for most Promise utilities and for ... of reduced the need for most iterable utilities.
4. It's far easier and more intuitive to iterate and manipulate when you can just use the same for, if-else, etc. that you already know well.
5. It would open up the door for substantial engine optimization:
   • A lot less indirection due to less chaining, fewer method calls, and more points where the engine can optimize directly.
   • The closure formed is not a full nor standard function, so engines can optimize for that (i.e. closure/argument pair may be passed in registers, and the return value is an optional completion)
   • The parallel keyword allows merging to be done without creating an entire Observable to do so. Instead, engines can simply emit directly from the parent context, making it faster to both create and execute.

If we use @ here, it's unavailable for future use.

Note that I just updated the proposal to simplify it some, and one of the changes involved using the keyword stream in place of that (and making it async-only).

Also, it's unclear how you'd complete a stream

Completing and erroring out of a stream would be via return and throw.

[dead-claudia]

Here's a gist comparing my syntactic idea to the existing status quo with RxJS, and here's another that is a simple utility library for observables à la Lodash/etc..

Incidentally, it actually seems make the code more verbose, but it's much clearer and explicit with its data flow, emphasizing it much more than the algorithmic steps. IMHO it's actually much more declarative in how it runs, since it admits some non-determinism naturally, and it provides declarative facilities to manage it. I'll admit it's not exactly based on a traditional von-Neumann-based model, and is more of a blend between FRP, data-flow programming, and stream processing, and would really show its true colors on the server, where you could avoid most of frameworks in general, and instead focus on just control flow. Here's my current server, adapted to full ES6, with one variant rewritten to use stream functions.

[benjamingr]

This needs to happen after es-observable lands. In the mean time, since Observable will implement Symbol.asyncIterator you will most likely be able to use for await loops until (and if) for... on loops land.

We need to make sure we're not spreading the proposal too thin.

[dead-claudia]

@benjamingr

This needs to happen after es-observable lands.

I'm okay with that (and was going with that assumption). It wasn't meant to block the current proposal, but more so focus on future directions. In particular, I was also leaving the door open for interop with Promises, which are a whole separate deal, potentially more complicating (that I intentionally omitted from this starter proposal). This is more like a stage -2 strawman, to get the idea out there, and there are still certain things I still have to resort to the Observable constructor over, such as flattening an Observable<Observable<T>> to an Observable<T> (monadic join) or zipping an Array<Observable<T>> to an Observable<Array<T>>.

you will most likely be able to use for await loops until (and if) for... on loops land.

That'd be nice, except for await ... of carries the wrong semantics (previous iterations asynchronously block subsequent ones, which is unsuitable for observation), so it'd only work for some use cases (like clicks and soft real-time push notifications).

Most likely, we'd continue using the same operator methods we use now, since it already helps the situation a lot (way easier to reason about than event emitters all over the place).

[dead-claudia]

Update: I've expanded on my concept here in this gist. It's blocked on Observables getting through, of course, but it's still somewhat related. It does in fact expand to integrate with both Promises and Observables, and I took caution to try to avoid certain performance cliffs in the process of explaining some of the details.

[aluanhaddad]

I really like the from syntax but, especially taken in the context of this statement

It's far easier and more intuitive to iterate and manipulate when you can just use the same for, if-else, etc. that you already know well.

I am inclined to ask a somewhat offtopic, but I think fair, question: I disagree that it is more intuitive, but for the sake of argument supposing it is more intuitive, it is . for and if, and if...else do not compose at all and employing them means going from a declarative to an imperative coding style when iterating and therefore I continue to use map and filter because I do not want to pre-create an empty result set call and then conditionally call push.

So my question is why is why is there so little (perceived) interest in adding some form of comprehension syntax to ECMAScript?

It would be highly valuable for observables, iterables, and plain old arrays for the language to support comprehensions such as in C#

from event in events 
where event != null
where event.Key == Key.Enter
from c in select event.InputSource.Value
select char.ToUpper(c)

and in Scala

for {
  (key, inputSource) <- events
  if inputSource != null
  if key == Key.Enter
  c <- inputSource.value 
} yield c.ToUpper

And many other languages. Perhaps ironically, while these languages have always had first class iterator consumption syntax, something ECMAScript did not have for arrays until ES2015, when someone needs to express something more complex than their fancier comprehension syntax allows for, they most often tend to fall back to APIs closely resembling those provided by ECMAScript arrays, such as Enumerable.Reduce and Enumerable.Any, rather than foreach/if.

Basically without a reified syntax for comprehensions, including both filtering and projecting within an expression context, these iterator-like syntactic addition offer very limited value.

I had heard that something like this was on the table at some point for ES2015 and was scrapped. This would provide of a lot of value in general and needless to say introducing syntactic sugar for observables, without even having basic filter support would be, from my point of view, a waste.

Sorry for taking this off topic. With all the new iterator-like syntax being discussed in this issue, I really wanted to comment.

Thank you.