Request for justifications and guidance on await in pipeline

[JAForbes]

@bakkot

I don't think this is the right thread to revisit await. As a committee member, though, I agree with the assessment that it's something this proposal needs to handle if it's to get through; if you want to discuss that further, I think a new issue would be appropriate.

I'd like to hear the reasoning for including await in the pipeline itself. I believe it's unnecessary, especially for an initial release and without a solid justification. We should seek to release a |> that only executes functions and leave ourselves room to add it if there's community demand for it in the wild.

I do not want to give the impression that this is an OO/FP schism. I like await, but I do not see it's usage in the pipeline as intuitive or helpful, and it's inclusion as affecting core functionality of the proposal.

I can easily see myself awaiting a pipeline await x |> f |> then (g) |> then (h). I Think that's quite intuitive.

And I can see myself awaiting within a function in a pipeline:

x |> async function f(y){ await ... } |> then g |> then h

But both these examples fit in with the existing design of the language, it meets expectations, it's intuitive, simple and flexible.

If I'm wrong, we can always come back and add await when we are all clear what the best way to do that turns out to be. But I see no reason not to move a simple pipeline forward.

84 proposes removing automatic function execution to accommodate the committee's request, but that makes the default usage awkward. And to propose such amendments simply to accommodate a need that I haven't heard a justification for, seems questionable.

[bakkot]

Sure, let me give my own thinking here.

Broadly, await can be naturally thought of as a way to transform one value into another (i.e., a promise to its resolved value), just like a unary function, even though it is significantly different from a unary function under the hood. As such it seems like a natural fit for the pipeline operator to me.

As an example, if I have a long chain of function applications mixed with awaits like

console.log(filter(parse(await fetch(extractRemoteUrl(await readDB(makeQuery('some query'))))));

then I would very much expect to be able to write something along the lines of

'some query'
  |> makeQuery
  |> readDB
  |> await
  |> extractRemoteUrl
  |> fetch
  |> await
  |> parse
  |> filter
  |> console.log;

since this is how I'd generally think about what I'm writing. Otherwise the pipeline proposal is effectively unusable in async contexts If we simply left out support for await altogether, I'd have to write

'some query'
  |> makeQuery
  |> readDB
  |> (_ => _.then(extractRemoteUrl))
  |> (_ => _.then(fetch))
  |> (_ => _.then(parse))
  |> (_ => _.then(filter))
  |> (_ => _.then(console.log))

or

'some query'
  |> makeQuery
  |> readDB
  |> (_ => _.then(extractRemoteUrl))
  |> (_ => _.then(fetch))
  |> (_ => _.then(_ => (_ |> parse | filter | console.log)))

or

(await fetch(
  getRemoteUrl(await ('some query'
    |> makeQuery
    |> readDB))
)
  |> parse
  |> filter
  |> console.log

or, indeed, any of several other broadly equivalent options, all of which feel like awkward ways of expressing the thing I actually want to express. I think the obvious way of writing that long chain of function calls as a pipeline - the only obvious way - is the version in which I await in the middle of a pipeline just as I currently await in the middle of a sequence of function applications.

To address your example,

await x |> f |> then (g) |> then (h)

would not work as currently specified: then is not a built-in global (nor could it be, I suspect, for web compat reasons), and in the absence of placeholders the pipeline syntax could invoke either bare functions (f) or bare methods (.then), but certainly not both. For this to work, you'd need to define a global then which looked something like const then = f => p => p.then(f);.

Moreover, even assuming a global then, every step in the chain following what was an await in the un-pipelined code would need to invoke then unless it intended to manipulate the promise itself. This eliminates most of the benefits of syntax support for async/await, which allows most programmers in most circumstances to forget about Promises altogether. (Sometimes engines too, for that matter.) It's also a lot of boilerplate of precisely the sort the async/await syntax is intended to cut down on.

I would object to the inclusion of any proposal which could not be used cleanly with async/await.

[jridgewell]

I think the obvious way of writing that long chain of function calls as a pipeline - the only obvious way - is the version in which I await in the middle of a pipeline just as I currently await in the middle of a sequence of function applications.

👍

For this to work, you'd need to define a global then which looked something like const then = f => p => p.then(f);

Higher order functions are cool, but definitely shouldn't be necessary to accomplish something so simple. Not to mention how difficult it'd be for a beginner to understand. I cannot imagine any beginner being able to think of this solution on their own.

Moreover, even assuming a global then, every step in the chain following what was an await in the un-pipelined code would need to invoke then unless it intended to manipulate the promise itself. It's also a lot of boilerplate of precisely the sort the async/await syntax is intended to cut down on

[JAForbes]

Thank you for the input please keep it coming.

For this to work, you'd need to define a global then which looked something like const then = f => p => p.then(f);.

It doesn't need to be global, it can be an import, it can be a static function on Promise. The point is, we can solve this with functions today.

every step in the chain following what was an await in the un-pipelined code would need to invoke then

True, but if you're writing async heavy code, then you'll likely be interspersing await throughout your pipeline as well.

Higher order functions are cool, but definitely shouldn't be necessary to accomplish something so simple.

1. I'd like some data on that assertion. then (f) vs f |> await, which is simpler to follow / replicate in their own code is an open question.
2. They don't need to define then themselves.
3. This feature enables function composition. Higher order functions are natural in this context.

I would object to the inclusion of any proposal which could not be used cleanly with async/await.

It appears including it is complicating an otherwise simple proposal. So even if you believe it needs to be supported, why can't we get a simple version into the language first, and add await afterwards?

[bakkot]

True, but if you're writing async heavy code, then you'll likely be interspersing await throughout your pipeline as well.

Yes. But it won't be on every step, just those which require awaiting (as in my example); moreover, the rest of my code (outside the pipeline) probably also already uses await, and probably does not already use then. So I don't think these are anything like equally bad.

It appears including it is complicating an otherwise simple proposal. So even if you believe it needs to be supported, why can't we get a simple version into the language first, and add await afterwards?

Because the ultimate method by which await is supported might well affect the overall design of the feature.

[JAForbes]

Just to reiterate, shipping a non awaiting supporting |> doesn't prevent us from shipping it later. And we can solve async code in userland e.g.:

So for now:

'some query'
  |> makeQuery
  |> readDB
  |> await
  |> extractRemoteUrl
  |> fetch
  |> await
  |> parse
  |> filter
  |> console.log;

Is simply:

import { then } from wherever

'some query'
  |> makeQuery
  |> then ( readDB )
  |> then ( extractRemoteUrl )
  |> then ( fetch )
  |> then ( parse )
  |> then ( filter )
  |> then ( console.log )

I understand a little better why the committee wants it supported. And I'm not even opposed to it. I'm opposed to it complicating a simple proposal and delaying it potentially unnecessarily.

I personally prefer the latter, and want to know I'm dealing with Promises. But I'm not all users, and that's fine. But can we get this into the language as is and solve this problem in a subsequent release? If not, why?

[JAForbes]

Because the ultimate method by which await is supported might well affect the overall design of the feature.

That's a sign to me, that we're approaching this the wrong way.

[JAForbes]

Yes. But it won't be on every step, just those which require awaiting (as in my example); moreover, the rest of my code (outside the pipeline) probably also already uses await, and probably does not already use then. So I don't think these are anything like equally bad.

I think its debatable, and again designing for existing usage not for future usage. Of course it uses await, await is already in the language.

[bakkot]

Just to reiterate, shipping a non awaiting supporting |> doesn't prevent us from shipping it later.

Sure it does. Or rather, to be precise, it prevents us from shipping some possible designs. For example, to support the syntax I'm using above we might (or might not) want to ban AwaitExpressions as the RHS of a pipeline; if we naively shipped |> without that restriction, we could not later add it in.

This is the primary experience of designing JavaScript: our past decisions constrain our future ones. Shipping an incomplete design often prevents us from shipping the design we'd ultimately prefer.

I'm opposed to it complicating a simple proposal and delaying it potentially unnecessarily.

It'll be years and years and years before you can use this feature on the web, assuming it even gets through committee; for the foreseeable future all use of this feature will be through Babel. But you can write your own Babel plugins, now, that do whatever you want! You don't need TC39 for that. I don't think this is a good argument for shipping a feature we have not entirely thought through, including its interactions with other parts of the language like await.

I personally prefer the latter, and want to know I'm dealing with Promises.

I also want to know I'm dealing with promises, but that's what using await does. (If you're a Haskeller, do you avoid using <- in preference for >>=? I don't, generally; I think <- is explicit enough and more clearly expresses my intent.)

Moreover, the version without then makes it explicit which functions do and do not return promises, since await is only used for those which do. By contrast, the version with then mixes them together and relies on the auto-flattening behavior of promises to hide the type confusion this implies. I hate that; this is part of the reason I find it important that programmers be able to consistently use await.

I think its debatable, and again designing for existing usage not for future usage. Of course it uses await, await is already in the language.

So is Promise.prototype.then, but I don't use it nearly as much. I don't think that I would stop using await if this proposal landed tomorrow.

You can't justify arbitrary designs by saying that they're designed for future usage. If you'd like to give a reason to believe that awaiting a value will be less useful in a future in which we have the pipeline operator, please do, but that's not something I currently expect.

[JAForbes]

Or rather, to be precise, it prevents us from shipping some possible designs.

In the linked tc39 discussion it was proposed await could be banned to allow for a release. So you could ship now, and add support later.

It'll be years and years and years before you can use this feature on the web, assuming it even gets through committee; for the foreseeable future all use of this feature will be through Babel. But you can write your own Babel plugins, now.

I hear this argument a lot, but it’s impractical. I do not want to use a babel plugin for a feature that is unlikely to ever land because then I’ve got legacy code for a language that doesn’t exist. If a basic version is on a standards track, I am more likely to take the chance.

By contrast, the version with then mixes them together and relies on the auto-flattening behavior of promises to hide the type confusion this implies. I hate that; this is part of the reason I find it important that programmers be able to consistently use await.

The behavior of Promises is orthogonal to your point, you can await non Promises, you can await undefined. If there’s any lesson to be learned from the auto flattening design decision it is to not bake in conveniences for the community without justification. In a similar sense, map/chain could have been implemented and released first, and then if the community wanted then, it could have been added in a subsequent release. It’s an ironic example.

So is Promise.prototype.then, but I don't use it nearly as much. I don't think that I would stop using await if this proposal landed tomorrow.

I do use Promise::then, but we’re both a sample of 1 so it’s neither here nor there.

You can't justify arbitrary designs by saying that they're designed for future usage

I’m justifying implementing a |> similar to almost every implementation that exists. Composition relies on simple pieces that do 1 thing, and combining them to create more complex functionality. Injecting await in a composition pipeline is a radical idea, and I’m not opposed to it but I find it perplexing that my design (which is in several languages) is arbitrary, but tc39’s isn’t.

Yes, I do think our industry is trending towards functional practices, and I do think composition of pure functions is the best solution we have to problems like concurrency/distributed system. And I'm confident that ideas that were very recently considered fringe, our now mainstream, and that trend will continue.

But even if you don't believe that, or see the relevance to this discussion - it doesn't matter, I'm proposing something proven should be shipped first. It's not an arbitrary design by any means. I'm looking for justifications for a constraint that is leading to designs like #84.

If you'd like to give a reason to believe that awaiting a value will be less useful in a future in which we have the pipeline operator

That’s not my aim at all. I’m not opposed to adding it. But I’ve just seen a proposal that takes the core proposition of composition and introduced new concepts based on a relatively new language to solve problems for users that haven’t requested them based on unproven needs. Such a proposal leads me to wonder, why are these constraints present in the first place? And if these constraints are forcing such radical changes to an already proven design, are the constraints justifiable?

But I really don’t want you to think I’m against adding await. I just want to make sure it’s not going to make FP language features awkward to use without justification. Your await proposal seems fine to me. But #84 does not. And the justification for that proposal is supporting await, hence this thread.

If we can add await without making composition awkward, I'm not opposed at all.

[kurtmilam]

I'm in strong agreement with @JAForbes .

I've been watching this proposal for several months. Over that time, it has seemed to me that discussions revolving around how (and whether) to force support for await into this proposal have far outnumbered discussions on all other topics.

It would be a real shame if adding support for await comes with it the requirements of a more verbose syntax and/or leads to a feature that is more difficult for runtime engines to optimize.

Removing a feature from a language is much harder than adding a new one. With that in mind, I am very strongly of the opinion that it makes sense to hold off on including support for await in the first implementation of the pipeline operator.

[ljharb]

Note that that argument is also one for not adding pipeline in the first place, until after figuring out the shape of await support.

[kurtmilam]

@ljharb

Note that that argument is also one for not adding pipeline in the first place, until after figuring out the shape of await support.

I would prefer no native pipeline operator to an overly complex operator that requires a tortured syntax and / or is difficult for runtime engines to optimize.

I would gladly continue to use existing implementations of pipe (e.g. Ramda's, Sanctuary's or my own) if that increases the likelihood that the implementation of the native operator finally arrives without extra baggage in the form of complexities of questionable utility that can never be discarded or improved upon.

[bakkot]

@JAForbes

In the linked tc39 discussion it was proposed await could be banned to allow for a release. So you could ship now, and add support later.

Only if we wanted to go with that one particular design, which, if we had already decided, we could just include in the proposal to start. That's not a general solution.

I hear this argument a lot, but it’s impractical. I do not want to use a babel plugin for a feature that is unlikely to ever land because then I’ve got legacy code for a language that doesn’t exist. If a basic version is on a standards track, I am more likely to take the chance.

That's understandable, but your reluctance really does not seem like a sufficiently good reason to rush an incomplete feature through committee to me.

The behavior of Promises is orthogonal to your point, you can await non Promises, you can await undefined.

I don't think it is orthogonal. If I'm using await, it's most natural to only await Promise-returning things. If I'm using .then, I'm forced to use it for everything after the first step which returns a promise.

Composition relies on simple pieces that do 1 thing, and combining them to create more complex functionality. Injecting await in a composition pipeline is a radical idea, and I’m not opposed to it but I find it perplexing that my design (which is in several languages) is arbitrary, but tc39’s isn’t.

I disagree strongly with the idea that it is more radical to include await than to leave it out. Prior art from languages which do not have similar support for await doesn't seem that relevant; if there's an example of a language which has a similar kind of await and also a built-in pipeline operator which does not allow you to await the value flowing through the pipe, that would be more interesting.

But even if you don't believe that, or see the relevance to this discussion - it doesn't matter, I'm proposing something proven should be shipped first. It's not an arbitrary design by any means. I'm looking for justifications for a constraint that is leading to designs like #84.

The core of my response continues to be what I gave in my first comment: because await may be thought of as a way of transforming one value into another, because await is an important part of the language yet code which mixes await and the pipeline operator is awkward (assuming it does not support it), and because shipping a design without support cuts off some possible future designs, I feel strongly that the pipeline operator ought to support await. That's the justification.

---

@kurtmilam

Over that time, it has seemed to me that discussions revolving around how (and whether) to force support for await into this proposal have far outnumbered discussions on all other topics.

I don't think I interpret this the same way you do. To me this says that there's a lot of demand for await support, not that it's something we can ignore without concern.

And I want to push back on the idea that await is being forced in. Like I say above, it seems like a very natural fit for the pipeline operator; I would be more surprised by its absence than its presence.

It would be a real shame if adding support for await comes with it the requirements of a more verbose syntax and/or leads to a feature that is more difficult for runtime engines to optimize.

While true, this is not an argument for ignoring the problem.

I think including support for await would almost certainly be more optimizable than requiring repeated thens, in the same way that await is currently more optimizable than .then. If you used the pipeline without either, I would expect there to be no impact on performance at all. I think the "difficult for runtime engines to optimize" thing is entirely a red herring.

Removing a feature from a language is much harder than adding a new one. With that in mind, I am very strongly of the opinion that it makes sense to hold off on including support for await in the first implementation of the pipeline operator.

As Jordan says, the second sentence here does not at all follow from the first. Just the opposite, in fact.

[zenparsing]

It's normal to feel disappointment and frustration when we feel like our point of view is being deprioritized. On the other hand, language design (especially for a lingua franca like Javascript) is a kind of public policy debate, where we must balance the points of view of all participants and strive for holistic, rather than pure, beauty.

In that sense, it's much more productive to focus on the actual harm that a proposed policy might cause (e.g. burdensome repetition of a binding token) rather than on abstract notions about who "wins" and who "loses".

[JAForbes]

@zenparsing I'm not interested in winning or losing at all. I hope that's being conveyed. I really just want this feature to be the best it can be. And I want it to be enjoyed by everybody. I think this feature will help me teach FP to a wider audience, it's a catalyst. I just want to be absolutely sure we're not sacrificing core functionality without being absolutely sure it's justified. But I'm not at all interested in winning.

does not seem like a sufficiently good reason to rush an incomplete feature through committee to me.

It's not really incomplete or rushed if it's already used in many other languages. It's just a commitment that composing functions is what |> does. Which isn't a big commitment.

I disagree strongly with the idea that it is more radical to include await

Just to clarify, when I said radical I meant it in the dictionary sense. As in, it's a new untested idea. Not as in it's a reckless or bad idea. I think it would probably be a good fit providing composing functions isn't hindered.

I don't think it is orthogonal. If I'm using await, it's most natural to only await Promise-returning things.

I think that's less true the moment you have |>. await enables a developer to turn a compositional style into a more procedural one. You get to use features like for and try and catch, and store things in variables. For certain types of work and contexts that is all very beneficial. I use await all the time.

But if you write programs that lean on composition you rarely have variables, you want to avoid for and try and catch, and being forced into an explicit data flow is a valuable constraint that leads to systems that are easier to reason about. 2 different styles with different benefits, but |> is an enabler of the latter style.

I program with pipe a lot, and when I find myself reaching for more context I make sure I can justify it to myself. I'm personally confident that leads to simpler solutions and less bugs. If it's not working in a particular situation I think that's a sign that I should go full imperative for that particular function. Mixing styles could work too, but I think it'll become quite common to have functions that are only pipelines and not much else.

That's not to say you can't mix the styles, that should be encouraged, but I don't want to sacrifice the style it does enable, for the benefit of a style it doesn't. We are adopting an existing feature, and that feature was designed with an explicit purpose, so I think it's reasonable to not sacrifice it's initial purpose when there's already affordances for other styles (like method chaining).

The core of my response continues to be what I gave in my first comment: because await may be thought of as a way of transforming one value into another, because await is an important part of the language yet code which mixes await and the pipeline operator is awkward (assuming it does not support it), and because shipping a design without support cuts off some possible future designs.

I think this is a good resource if this question comes up in the future. I don't agree with your justifications but I don't think they're unreasonable either. I'm just happy to have an answer. Thank you for everybody's time. I'm going to close this issue now, but please feel free continuing the discussion if there's more to say.

[TehShrike]

Was it actually established that await support is required for this proposal to move forward?

The TC39 discussion notes didn't indicate that the group thought it was a requirement.

I know some people here in the issues consider await important, but I think we should establish that the TC39 body considers it a requirement before we give in to unbounded bikeshedding.

I suggest that this issue be re-opened until someone can point to some kind of consensus that this feature request is actually blocking, or we determine that it is not blocking.

@littledan you're the champion, have any thoughts?

[mAAdhaTTah]

@TehShrike The conclusion was reached above; @bakkot is a member of the committee.

[TehShrike]

I recognize/respect that, but one committee member's voice is not consensus. The discussion at the last meeting would seem to put him in the minority of committee members.

[ljharb]

Consensus typically means that every member must agree.

[TehShrike]

Are you implying that if one TC39 member gives the thumbs-down to a proposal, it won't move forward?

I don't know enough about TC39 policies in this area.

I assumed that if a majority of members were against adding await, it would be appropriate to advance the feature without await.

If a minority of members want await, and a majority don't want await, does that mean that the proposal can't move forward at all? Does it imply that the proposal must/must not include await to progress?

[ljharb]

Yes, that’s what consensus means - that any one member can block anything (although often, if only one member is dissenting, they may choose to withdraw their objection).

So, if even one member thinks the proposal must include await to proceed, then it can never proceed while that is true and it lacks await.

(To clarify, @bakkot is not the only member who thinks await is a requirement here in some form; i share his concerns.)

[TehShrike]

I'm not familiar with TC39's inner workings, so I appreciate the explanation.

From what you're saying, I assume that if there are committee members who think that await is too complex/undesirable, then the proposal can't proceed with the feature either?

I'm most concerned about deadlock, I don't want to miss out on this language feature because no good compromise exists :-x

---

Maybe it' would be useful to frame the discussion in the other issues as "is it possible to alter the proposal in a way that satisfies both groups (anti-complexity and pro-async-feature)."

If not, I suppose the group will either settle for a complex pipeline operator, a pipeline operator without async, or the proposal will die in deadlock.

[js-choi]

@TehShrike: The TC39 discussion notes didn't indicate that the group thought it was a requirement.

Was the current pipeline proposal discussed at the most recent TC39 meeting, the 62nd in late January 2018? The 62nd’s agenda does not include an item for the proposal, and the 62nd’s notes have not yet been published in the usual repository for meeting notes.

For reference, here is the relevant discussion from the 61st TC39 recent meeting in November 2017. The discussion was largely exploratory in nature, went over time with no conclusion, and was deferred to the overflow agenda. There isn’t any more recorded discussion for the 61st.

For what it is worth, the 61st’s discussion occurred before parameterized pipelining was first proposed (by @gilbert in #75 and by @zenparsing in #84). Adding parameterized pipelining as an additional operator from the current proposal’s tacit pipelining—like how Clojure separates them into ->, ->>, and as->—is still an important new idea (see https://github.com/tc39/proposal-pipeline-operator/issues/84#issuecomment-359939771, https://github.com/tc39/proposal-pipeline-operator/issues/75#issuecomment-360551654, and https://github.com/tc39/proposal-pipeline-operator/issues/75#issuecomment-361708340). This model is both simple but versatile, and it covers the await use case without making a special case just for await; yet it also does this while also keeping tacit pipelining as separate operators, accomodating point-free functional programming. This dual model will hopefully be considered the next time TC39 discusses it: the 63rd meeting in late March. For now, however, pipelining is not on the 63rd’s agenda at all yet.

[bakkot]

To be clear, this isn't something I'd block consensus for, but without wanting to speak for anyone else I would guess that much of the rest of the committee would similarly feel that interaction with the rest of the language, including await, is something we'd want worked out before putting this in the language.

[littledan]

No, it wasn't established in committee; earlier comments I made were based on out of committee discussion, and no one threatened to withhold consensus. However, we are still pretty early in the process. I think now is a good time to focus on gathering requirements. Await is frequently requested and makes logical sense here, so I'd like to support it unless there is a strong reason not to (e.g., if it were impossible without tons of complexity and problematic edge cases).