Discussion: Proposal Mixes

gilbert commented 6 years ago

Hi all, I've just updated the wiki with the current status of all proposals so far. This thread specifically is for exploring the possibility of mixing together the two main proposals (F-sharp style and Hack style). Please visit and read that page for details.

mAAdhaTTah commented 6 years ago

They wouldn't. It's to avoid the "do I use ■ or ▲ here?" problem. It also keeps one explainable as the other.

rbuckton commented 6 years ago

I'm not certain that Proposal 4 is tenable in the long term as it suffers from the same "garden path" issues that Waldemar brought up for partial application. In general I think that using a placeholder in an arbitrary expression would be a very small corner case for this feature that can already be addressed using either () or arrow functions. I strongly believe that combining F#-style pipes with partial application provides two very specific features that can be learned in isolation with clear rules and semantics:

v |> F roughly means F(v)
F(?) roughly means x => F(x)

For the corner cases, there are always existing behaviors to fall back on:

(v |> F).prop
(v |> F)[expr]
(v |> F)()
await (v |> F)
yield v |> F
typeof (v |> F)
v |> (x => expr)
etc.

Grouping with () for operators like await, yield, etc. still maintains left-to-right evaluation order. For cases where you need to use the piped value in an arbitrary expression you can fall back to arrow functions.

F# pipelines (and partial application) also give us more future flexibility with things like function composition. Imagine if we add f ~> g to mean x => g(f(x)):

anArrayOfArrays  |> flatMap(?, pickEveryN(?, 2) ~> kebabCase)

// which is roughly the same as:

flatMap(anArrayOfArrays, x => kebabCase(pickEveryN(x, 2)));

js-choi commented 6 years ago

I’m going to start abbreviating “partial application” as PA below.

@mAAdhaTTah: It also keeps one explainable as the other.

Explaining pipe placeholders in terms of unary function calls on partially applied functions…Well, it comes around again to how desirable that goal really is. It strikes me as a lost cause, especially with all the other types of non-function-call expressions.

@mAAdhaTTah: It's to avoid the "do I use ■ or ▲ here?" problem.

As for “do I use ■ [pipe placeholder] or ▲ [PA placeholder] here [in a pipe]?”, the answer I think could be pretty simple, even for Proposal 4. “You always use ■ in a pipe, e.g. x |> f(■, 3) and never x |> f(▲, 3)—unless you’re calling a named unary function f, in which case you may use the tacit call x |> f (which would never include ▲)—or in the rare case that you actually want to have the pipe’s result be a PA function, e.g., f(x, ▲), in which case you’d use both ■ and ▲, e.g. x |> f(■, ▲).”

@rbuckton: In general I think that using a placeholder in an arbitrary expression would be a very small corner case for this feature that can already be addressed using either () or arrow functions.

First, @rbuckton, I want to thank you for all the hard work you’ve put into your PA proposal, as well ES/JS in general.

I don’t know if this is the best issue to discuss how important supporting arbitrary expressions in pipeline RHSes is, since this issue is specifically about Proposals 3 and 4…

But I do want to push back on this specific statement—particularly the “very small corner case” part. In fact, I would say that it is possible that non-function-call expressions could end up being the majority of cases of a pipeline operator that supported them.

In particular, property/method calls, arithmetic expressions, await, typeof, and instanceof are all very frequent and valid transformations on values. A syntax that requires the writer to write and the reader to read something like:

(
  await (
    await (
      (x + 3) * 2 + await f(x + 3)
    ) |> g
  )
)[Symbol.iterator] instanceof Function

…instead of…

x
  |> ■ + 3
  |> ■ * 2 + await f(■)
  |> await ■
  |> g(■)
  |> await ■
  |> ■[Symbol.iterator]
  |> ■ instanceof Function

…is not desirable. Isolating pipelines within parentheses within usual expressions, such as what you suggest, compromises the benefits, the same benefits that it would bring to function calls.

In particular, being able to chain property/method calls, instanceof, and typeof would be a huge boon, because they are very common in code (including in functional programming)—and formatting them in long expressions is difficult, as may be seen above.

As for the garden-path problem, that would apply only to Proposal 4 and not to Proposal 2 or 3. But even for Proposal 4, as long as the reader does not have to backtrack in order to interpret a pipe’s RHS correctly, there isn’t really a garden-path problem. As long as the rule for the tacit call is simple and would fail early in non-function-call expressions, then little backtracking would be needed from either the compiler or the human reader.

(For instance, why couldn’t Proposal 4 support tacit function calls only for bare identifiers in the RHS, like x |> f, requiring the programmer to write x |> f(3, ■), and making x |> f(3, ▲) an error? After all, PA/▲ is useful in a tacit-call pipe’s RHS only insofar that the RHS is a unary function, i.e., only insofar that it would be equivalent to using pipe placeholders / ■ instead.)

As for function composition, function composition is a simple binary operation that does not involve placeholders at all, unlike PA. I’m uncertain how the pipe placeholders in Proposals 2/3/4 could conceivably affect a binary function-composition operator.

In any case, I do sympathize with the desire to make point-free functional programming in JS more ergonomic. I hope that some of these arguments—that non-function-call expressions are useful to pipe values through too, and that they can conceivably coexist with tacit function calls in a versatile but simple way.

I haven’t had time to update the wiki or finish the table of possible interactions between pipe placeholders and syntactic PA, as promised. I’ll try to finish it tomorrow as free time allows. My apologies.

rbuckton commented 6 years ago

x
 |> ■ + 3
 |> ■ * 2 + await f(■)
 |> await ■
 |> g(■)
 |> await ■
 |> ■[Symbol.iterator]
 |> ■ instanceof Function

This could also be written as:

await x
  |> (x => x + 3)
  |> (async (x) => x * 2 + await f(x))
  |> (async (x) => await g(await x))
  |> (async (x) => x[Symbol.iterator] instanceof Function);

As far as operators are concerned, I'd love to see higher-order variants of operators like the ones in this strawman: https://github.com/rbuckton/proposal-functional-operators.

I'd like to spend some time this week thinking about this example and how it would apply to F# style pipes, PA, higher-order operators, and function composition (e.g. f ~> g). The only thing that is missing is a clear way of handing await/yield/yield*, since those operators currently cannot be made higher order.

js-choi commented 6 years ago

@rbuckton: Please do think about how your functional-operators proposal might fit into pipelines. It is very interesting, and I’m a fan of it. It would be very powerful for much terse point-free, tacit functional programming; I would use it often, just like I would use your other proposals often. I hope that it will be championed by someone and considered by TC39. However, it is uncertain to what extent we may coordinate the pipeline operator’s design with functional-operators, as the latter has no champion yet.

This coupling of pipelines to syntactic PA, composition, and functional operators is very complicating—perhaps unnecessarily so, even for tacit-style programming. Operators are simply not the same as functions. There will always be some impedance mismatch when trying to map operators onto functions, because many operators have special rules that cannot be explained by simple function calls. This includes await, yield, and yield * today; there may be more in the future. If homoiconic, syntactically uniform macros existed à la Lisp, maybe things would be different. I don’t know.

We talk about future proofing, but there is another thing here for which we should also future proof: future operators. It may be in fact most future proof for a pipe to allow arbitrary expressions, which would accommodate expressions of any sort.

These non-function-call expressions are not an edge case. They are a large segment—maybe even the majority—of expressions that a JS programmer may write, even when using tacit functional style. Just look at optional method chaining, which is essentially chaining together a binary operator ?? with a method call. There are numerous other examples.

Look also at the verbosity of the example above of a pipeline with many non-function-call steps, refactored to use arrow functions. Either the writer must bunch many data-transformation steps together into the bodies of arrow functions and nullify the benefit of separation of steps in a pipeline, or they may place each data transformation in its own step but with an (async x => …) surrounding each one, and place an await at the very beginning only if a promise is being threaded through the pipeline, and write far more parentheses. The reader is required in turn to read the data-transformation steps through that visual noise…

…And the optimizer now must determine how to inline these nested await functions, or else it will unnecessarily allocate and garbage-collect memory for each async function for each time the pipeline is evaluated, especially in hot loops—as well as wrapping, passing through, and unwrapping unnecessary promises for each step, when their awaits could have been evaluated in the surrounding async function the whole time.

The garden-path problem and the which-placeholder-do-I-use problems might be obviated through bikeshedding the precise rule of Proposal 4’s tacit-call mode. It could be a false dilemma between giving up on Proposal 4 and giving up on PA syntax. And even choosing between Proposal 4 and Proposal 1 is a false dilemma; there is always Proposal 3, with its own cost (one more operator) that might be minimizable through further bikeshedding.

And binary function composition is still supportable by Proposals 2, 3, and 4; in fact, Proposals 2, 3, and 4 might make binary function composition more useful, by enabling its use in pipelines for creating functions.

In any case, before any proposal is determined to be untenable, its particulars should be bikeshedded until all possibilities are exhausted. That table of proposal interactions that I’ve been writing will hopefully concretely clarify the space of possibilities.

But I do await your further refactoring of the example above to use higher-order operators and function composition. I’m a big fan of your proposals.

I simply wish to point out that such an example might not be mutually exclusive with pipe placeholders, which would serve many not-edge case. I would find pipes very useful for tacit programming styles myself, but I don’t want it to be useful only for tacit style. I see an easy way for these pipes to make much terser and readable the do expressions or the sequentially defined variables that I would have had to write/read anyway.

mAAdhaTTah commented 6 years ago

Minor note: I wrote and posted this before seeing @js-choi's response above.

@rbuckton Minor note: your alternative would need some await calls in between the async functions in the pipeline, as x comes into them as a promise.

The current proposal for await with F#-style is specced in #85, but hasn't been merged.

We were chatting about the pipeline operator on Twitter, and Henry Zhu pointed out that babel implementations of the various proposals could be helpful for us. I'm working on securing some time for OS work from my current employer and may be able to dedicate some time to this.

Would working implementations for the current proposals be helpful? I think the current pipeline plugin in the babel repo matches the current spec, so we'd just need to add await support for it to match the latest F# proposal. It looks like we're mostly debating that and some form of a Smart Mix, so I'd do that one after. Lastly, I don't think there's a babel plugin for partial application, so I'd be happy to work with @rbuckton on that as well. We could also throw in a functional operators plugin if we have time / inclination.

Relatedly, I'd like to propose dropping some of these proposals. Specifically, Proposal 3 seems to have gotten zero traction; we only discuss it in the context of "2/3/4" rather than arguing for it individually, so I think we can kill it.

Additionally, I think we should combine 2 & 4 into a single proposal, taking 4 as it currently stands and answering the last "current debates" question by requiring await / yield / etc. to use the placeholder. We end up essentially with Hack-style allowing unary functions without a placeholder.

That would mean we'd only need to build 3 plugins (F#+await, Hack+unary, and a partial app plugin), and see how they play together. Hack+unary & partial app plugins will allow the token to be configurable.

Thoughts?

I want to thank all of you for working so hard on this and being so professional throughout. We obviously disagree, but we've managed to come pretty far without getting personal. I think we've got a lot of momentum here, and we seem to be coalescing around these two alternatives, so I'm hoping these plugins can get us through this last decision and get us to consensus.

Jus to throw out something I've been mulling over: if we go with a proposal that comes with placeholders built-in, I expect using placeholders is going to be idiomatic for pipelines; that you're simply not going to see people using arrow functions there at all. If we're considering Hack-style, then besides the unary exception, I don't think we need to further optimize for F#-style because it won't be that common.

zenparsing commented 6 years ago

@mAAdhaTTah I'm not comfortable letting go of option 3 just yet. I think it is still gives a good combination of expressivity and explicitness, with room for bikeshedding.

js-choi commented 6 years ago

@mAAdhaTTah: We were chatting about the pipeline operator on Twitter, and Henry Zhu pointed out that babel implementations of the various proposals could be helpful for us. I'm working on securing some time for OS work from my current employer and may be able to dedicate some time to this.

I’d be open to attempting a Babel implementation of Proposal 4, although it would be my first one, and it would be dependent on my free time. The answers to certain unresolved bikeshedding questions (e.g., “What should the prototype’s placeholder token?” “What precisely is the RHS rule for parsing the pipe as a tacit function call?”) should be easily changeable in the Babel prototype.

I wonder if I should stop work on that table of possible interactions between the proposals in lieu of the concrete Babel prototypes.

Relatedly, I'd like to propose dropping some of these proposals. Specifically, Proposal 3 seems to have gotten zero traction; we only discuss it in the context of "2/3/4" rather than arguing for it individually, so I think we can kill it.

Additionally, I think we should combine 2 & 4 into a single proposal, taking 4 as it currently stands and answering the last "current debates" question by requiring await / yield / etc. to use the placeholder. We end up essentially with Hack-style allowing unary functions without a placeholder.

The nomenclature of Proposals 2 and 3 might still be useful during discussion, even if they would not be worth prototyping in Babel. Having said that, yes, it does seem like there is relatively little buy in for either 2 and 3, although they remain as theoretical options should the competing constraints on 4 become too tight. I agree with @zenparsing that Proposal 3 still would not be a bad outcome.

I want to thank all of you for working so hard on this and being so professional throughout. We obviously disagree, but we've managed to come pretty far without getting personal. I think we've got a lot of momentum here, and we seem to be coalescing around these two alternatives, so I'm hoping these plugins can get us through this last decision and get us to consensus.

I very much concur. Thank you, everyone.

Jus to throw out something I've been mulling over: if we go with a proposal that comes with placeholders built-in, I expect using placeholders is going to be idiomatic for pipelines; that you're simply not going to see people using arrow functions there at all. If we're considering Hack-style, then besides the unary exception, I don't think we need to further optimize for F#-style because it won't be that common.

Yes, agreed; this is a point similar to one that I poorly tried to express above. Tacit calling would be useful only for unary functions anyway, and that applies too to tacit calling of PA functions (except in the rare circumstance that the desired result is a function value).

For instance, using the ■/▲ notation from above, there is the option of keeping x |> fn(▲, 2) as a valid tacit call on a PA function (i.e., fn(▲, 2)(x)), which would therefore be equivalent to x |> fn(■, 2). But there is also the option of forbidding any RHS without a ■ only unless the RHS is a simple bare identifier, in which case it would be a tacit function call (i.e., only tacit calls of the form x |> identifier would be allowed). This second option would make only x |> fn(■, 2) valid and turn x |> fn(▲, 2) into a syntax error, obviating the writer’s paralysis of choice. The second option would still make F-sharp style with PA ergonomic to write and read, e.g., x |> addOne—as long as the PA functions are bound to variables in preceding statements (e.g., const addOne = add(▲, 1)), as I believe they usually are—otherwise, one would simply use forms like x |> fn(■, 2).

mAAdhaTTah commented 6 years ago

@js-choi If you're willing to pick up Proposal 4, I could focus on 1 + partial app, and we have the bandwidth, we could get Proposal 3 in there as well, since we're not completely willing to kill it yet. This is also my first babel plugin, so definitely interested in collaborating. My email is in my GH profile, shoot me an email and let's get this going!

js-choi commented 6 years ago

@mAAdhaTTah: Done. My free time will be variable but I will do my best to keep public progress going. Thanks again.

littledan commented 6 years ago

Trying multiple options out in Babel and collecting more feedback that way sounds like a perfect plan to me. Some other things we might want to do two of:

Spec text drafts: Making sure something can be cleanly specified helps in the design project, e.g., to work out edge cases. It's also likely to be easier for a larger group of people to review than the actual implementation in Babel. Would you two be interested in writing spec text for these options? I'm happy to help with any questions you have.
Presentations to TC39: I removed the pipeline presentation from the previous TC39 because of the lack of a solid placeholder story. At the next TC39 meeting, I could present again on the two alternatives that you are discussing, their concrete details and the tradeoffs, and our plan to get community feedback based on Babel usage. This would just be a status update presentation; in my opinion, we should decide between the two options before advancing to Stage 2.

mAAdhaTTah commented 6 years ago

@littledan I have not written a spec text before, but first time for everything! Feedback from the committee on the two proposals would also be really helpful.

in my opinion, we should decide between the two options before advancing to Stage 2.

💯

js-choi commented 6 years ago

@littledan: I would be honored to write a specification draft for Proposal 4. Would it be better to finish writing the formal specification before finishing the Babel plugin?

mAAdhaTTah commented 6 years ago

@js-choi My thinking was writing the babel plugin would help with the spec. I've written code before, haven't written a spec, so figured writing the code would help me understand how to write the spec.

littledan commented 6 years ago

@js-choi @mAAdhaTTah I'm very happy to have your help on spec writing here. Feel free to ask any questions either here or on a PR. One last request: Please fill out this form to license any IP rights attached to these spec documents to Ecma, so that TC39 can consider them for inclusion in the final specification.

rbuckton commented 6 years ago

So as not to sidetrack this discussion, I've added my thoughts around some additional exploration for F#-style pipelines to #93.

mAAdhaTTah commented 6 years ago

@littledan You have any "How to write a spec"-type resources? Once I get to the spec writing, I'm also going to dive into other specs, see how they're written, but any additional resources you have would be helpful.

littledan commented 6 years ago

Some resources for starting out:

kurtmilam commented 6 years ago

@js-choi

(
  await (
    await (
      (x + 3) * 2 + await f(x + 3)
    ) |> g
  )
)[Symbol.iterator] instanceof Function

This example appears fantastical to me. I appreciate that it's not always trivial to come up with terse and meaningful toy examples. At the same time, I hope the consideration of examples the likes of which are highly unlikely to ever be encountered in the wild doesn't play an outsize role in determining the final implementation of this proposal.

These non-function-call expressions are not an edge case. They are a large segment—maybe even the majority—of expressions that a JS programmer may write, even when using tacit functional style.

It's trivial to write reusable utility functions to wrap a number of non-function-call expressions (or simply borrow existing functions from an established library, like Ramda, et al.).

For this reason, and with a few notable exceptions, I think it's easy to overstate the negative impact on ergonomics the community must expect to live with in order to use non-function-call expressions in conjunction with a pipeline operator implemented according to the F# proposal.

gilbert commented 6 years ago

I think it's easy to overstate the negative impact on ergonomics the community must expect to live with in order to use non-function-call expressions in conjunction with a pipeline operator implemented according to the F# proposal.

To be fair, you could say the same flipped around. According to the Hack-style proposal, it's easy to overstate the negative impact on ergonomics for using curried functions, which is the only significantly impacted use case.

kurtmilam commented 6 years ago

To be fair, you could say the same flipped around. According to the Hack-style proposal, it's easy to overstate the negative impact on ergonomics for using curried functions,

I see this as an apples to oranges comparison. There are techniques available to minimize the negative impact of using non-function-call expressions with an F# style pipeline operator. There are no such techniques available to minimize the negative impact of using unary functions with a Hack style pipeline operator.

which is the only real impacted use case.

Not just curried functions, rather the ergonomics of all unary functions are impacted.

Partial application via arrow function:

const parseInt10 = x => parseInt(x, 10)

Userland unary functions:

const inc = x => x + 1

Native unary functions:

Array.prototype.concat

It can get noisy:

const a = [0,1,2]
// F-Sharp style
'02'
  |> parseInt10
  |> inc
  |> a.concat

// Hack style
'02'
  |> parseInt10(■)
  |> inc(■)
  |> a.concat(■)

mAAdhaTTah commented 6 years ago

The current smart mix allows unary functions without a placeholder.

kurtmilam commented 6 years ago

I'm still ruminating over the Smart Mix. I like it from a usability and functionality standpoint. I do see the benefit in allowing non-function expressions on the RHS of the pipeline operator.

I'm a little concerned that Smart Mix will be difficult to specify and difficult for engines to implement.

My initial inclination is to prefer the F# style for the first iteration of the pipeline operator, hopefully specified in such a way that it could be expanded to either the Split Mix or Smart Mix style in a future iteration, giving developers time to work with the simplest possible version of the solution before bolting additional features (and complexities) on top.

We may find that developers are largely happy with the F# style. It's also possible that other proposals (like @rbuckton 's Partial Application and/or Functional Operator proposals gain traction in the meantime, and that those plus the F# style pipeline operator cover everyone's needs.

I'm not against the Smart or Split Mix styles, on principle. I am much less a fan of the Hack style, however.

gilbert commented 6 years ago

Not just curried functions, rather the ergonomics of all unary functions are impacted.

You caught my comment in between the 30 seconds I edited it :) I meant to say significantly impacted.

I don't think it's fair to call x |> f(■) noisy after saying it's easy to overstate how unergonomic non-function-calls are, e.g. x |> (a => a.map(f)). The former is less noisy than the latter.

Also, I would hope the pipeline op is ergonomic without having to write your own series of partially-applied functions like parseInt10 or importing a library with pre-curried functions.

js-choi commented 6 years ago

@kurtmilam: This example appears fantastical to me. I appreciate that it's not always trivial to come up with terse and meaningful toy examples. At the same time, I hope the consideration of examples the likes of which are highly unlikely to ever be encountered in the wild doesn't play an outsize role in determining the final implementation of this proposal.

Yes, my apologies. I acknowledge that it was a contrived example for demonstrative purposes. I have not yet had time to mine the existing corpus of JavaScript code to find real-world example uses of non-function-call expressions. It is my hope that I will be able to do so in the future, to better compare the different proposals’ ergonomics. Having said that, I hope that you will agree that non-function-call expressions remain a major use case, if not the majority, of data transformations in the current JS corpus. Whether that status quo should change is a separate question.

I'm a little concerned that Smart Mix will be difficult to specify and difficult for engines to implement.

I and others will do my best to specify and prototype Proposal 4, as well as the other pipe proposals. Its simplicity or complexity may be better judged, once concretized in that manner. I don’t think what I have in mind is very complex (tacit unary function calling when the RHS is a bare identifier; a nested do expression otherwise), but we will see. I am always open to changing my mind.

However, with regard to difficulty of engine implementation, I will add that I am concerned about the complexity of engine-level optimization of Proposal 1: F-sharp Style Only. This is particularly worrisome for unnecessary function generation and unnecessary promise wrapping. The more complex the RHS’s wrapper function becomes, the less likely it may become efficiently inlined. And optimizing unnecessary promise allocation / garbage collection is another can of worms.

kurtmilam commented 6 years ago

You caught my comment in between the 30 seconds I edited it :) I meant to say significantly impacted.

It's not just curried functions that are impacted. It is, instead, all unary functions that are impacted.

I don't think it's fair to call x |> f(■) noisy after saying it's easy to overstate how unergonomic non-function-calls are, e.g. x |> (a => a.map(f)). The former is less noisy than the latter.

x |> f(■), while significantly quieter than x |> (a => a.map(f)), is not equivalent to it.

Pointing out the unavoidable noisiness of one solution (Hack style with unary functions) is in no way minimizing the unavoidable noisiness of another solution. You brought up curried (actually unary) functions, and I was responding directly to that context.

Speaking of unavoidable noisiness, map is another function that's trivial to either write on your own or borrow from someone else, such that your example could be rewritten x |> map(f) in F# style, quieter than the equivalent x |> ■.map(f) in Hack style.

Also, I would hope the pipeline op is ergonomic without having to write your own series of partially-applied functions like parseInt10 or importing a library with pre-curried functions.

I share your hope, but I'm not convinced that we have to specify the final, be-all, end-all of pipeline operators from the beginning, rather than working our way towards that goal via an iterative process.

mAAdhaTTah commented 6 years ago

The more complex the RHS’s wrapper function becomes, the less likely it may become efficiently inlined. And optimizing unnecessary promise allocation / garbage collection is another can of worms.

I'm not sure why this would be true. My intuition is you could write code that desugars to the same thing in either style, and that optimizations that apply to one would apply to the other. Why would one be more optimizable than the other?

js-choi commented 6 years ago

@jschoi: The more complex the RHS’s wrapper function becomes, the less likely it may become efficiently inlined. And optimizing unnecessary promise allocation / garbage collection is another can of worms.

@mAAdhaTTah: I'm not sure why this would be true. My intuition is you could write code that desugars to the same thing in either style, and that optimizations that apply to one would apply to the other. Why would one be more optimizable than the other?

I agree that the claim of my first sentence in your quotation is more tenuous. As long as RHS inlining is possible, then Proposal 1 might be a zero-cost abstraction. But this becomes more complicated with nested async functions. Look at the second code block in https://github.com/tc39/proposal-pipeline-operator/issues/89#issuecomment-363660518. It may be considerably more complex for an engine to determine how to inline those inner async functions, which are now passing wrapped promises between each other, and which will need to have one extra await. But I would be happy to wrong about this.

kurtmilam commented 6 years ago

I hope that you will agree that non-function-call expressions remain a major use case

While I do agree with this statement, I'm not convinced that we have to begin with a solution that supports non-function-call expressions on the RHS from day one, especially if we can start with a less complex implementation that's specified in such a way that it is given room to grow based on feedback informed by actual developer experience, and especially since there already exist working solutions to the problem that can be employed in the interim.

The more complex the RHS’s wrapper function becomes, the less likely it may become efficiently inlined.

I'm not an expert in engine-optimization by any means, but it is my understanding that native compose and pipeline operators (even using the F# style) offer engine-writers significant opportunity for optimization via inlining.

I'd also like to point out that there exist common userland solutions that can be employed to minimize garbage collection in pipeline chains in the hot path.

// Don't define functions inside the hot path!
// This is good advice everywhere, not just inside a pipeline chain.

// Don't do this in the hot path:
x |> (x => parseInt(x, 10))

// Rather, define your function outside of the hotpath
const parseInt10 = x => parseInt(x, 10)

// Then use it in the hot path
x |> parseInt10 

// I think this is a well-known principle in JavaScript, already - nothing new here.

Even if your concerns about the inlinability of complex functions are justified, we have tools at our disposal to help us avoid the problem of generating excessive garbage in the hot path, not to mention the option of avoiding the operator altogether in the hot path if performance concerns prove to be well-founded.

gilbert commented 6 years ago

x |> f(■), while significantly quieter than x |> (a => a.map(f)), is not equivalent to it... You brought up curried (actually unary) functions, and I was responding directly to that context.

You originally brought up non-function-calls, and I was responding to that context. The point of your comment was essentially "it's trivial to delegate to user-land solutions for non-function-calls", suggesting users write their own utility functions or import existing libraries, and that these solutions are not unergonomic enough to warrant a different proposal.

To that point specifically I am arguing that you can flip it around from the perspective of the Hack-style proposal. It's trivial to write f(■) instead of f for the one use case (unary functions) that the Hack-style proposal negatively impacts. For all other use cases, it's more ergonomic.

kurtmilam commented 6 years ago

Look at the second code block in #89 (comment). It may be considerably more complex for an engine to determine how to inline those inner async functions

Notes:

You're reaching back to your admittedly contrived example for support. I've not come across any real-world use of await that looks anything like your example.
It's my understanding (although I may be oversimplifying) that await can be desugared into Promises and Promises can be desugared into callbacks. If my understanding is correct, I don't think nested awaits are going to be the problem you think they may be, but I could be wrong.

A relevant quote from Oliver Wendell Holmes, Jr.:

"Great cases like hard cases make bad law. For great cases are called great, not by reason of their importance in shaping the law of the future, but because of some accident of immediate overwhelming interest which appeals to the feelings and distorts the judgement."

I don't think it makes much sense to base important, irreversible decisions on fantastically extreme cases, especially when we could start with a simpler solution to which features (and concurrent complexity) can be added in the future, based on feedback informed by real-world experience.

kurtmilam commented 6 years ago

You originally brought up non-function-calls

@js-choi brought up non-function-call expressions, and I was responding to them.

As I mentioned earlier in our conversation, I join you in support of a solution that doesn't require external libraries or home-brewed utility functions, but I'm not certain that we need to nail down the specifics of that solution from the outset, rather than working towards it incrementally, and of the four proposals on offer, Hack style is my least favorite.

mAAdhaTTah commented 6 years ago

While I do agree with this statement, I'm not convinced that we have to begin with a solution that supports non-function-call expressions on the RHS from day one, especially if we can start with a less complex implementation that's specified in such a way that it is given room to grow based on feedback informed by actual developer experience, and especially since there already exist working solutions to the problem that can be employed in the interim.

I don't think an F#-style proposal being accepted would eventually get Hack-style pipelining. What I think is far more likely are new language features, like partial application & functional operators, that make the pipeline operator more useful over time. I think this is a strength of F#-style, in a "Unix Philosophy" kind of way.

js-choi commented 6 years ago

@kurtmilam: As I mentioned earlier in our conversation, I join you in support of a solution that doesn't require external libraries or home-brewed utility functions, but I'm not certain that we need to nail down the specifics of that solution from the outset, rather than working towards it incrementally…

@mAAdhaTTah: I don't think an F#-style proposal being accepted would eventually get Hack-style pipelining. What I think is far more likely are new language features, like partial application & functional operators, that make the pipeline operator more useful over time. I think this is a strength of F#-style, in a "Unix Philosophy" kind of way.

I sympathize with the desire to do incremental enhancement, rather than a perfect up-front design. The problem with incremental enhacement using Proposal 1: F-sharp Style Only is maintaining forward compatibility with a possible future Proposal 4: Smart Mix (or else pre-rejecting any future possibility of a Proposal 4). This might be a solvable problem. For instance, if a Proposal 1 required its RHS to be bare identifiers (for function values) and forbade more complex RHS expressions, then a Proposal 4 could easily be retrofitted onto it later. However, if the Proposal 1 allowed its RHS, then that would preclude any future Proposal 4, though a Proposal 3: Split Mix might still be possible.

If TC39 ends up choosing a form of Proposal 1, in the name of incremental enhancement to the language, then I would hope that it would at least keep the doors to Proposals 3 or 4 open, by limiting its pipe operator’s permitted RHS forms.

@kurtmilam: You're reaching back to your admittedly contrived example for support. I've not come across any real-world use of await that looks anything like your example.

I don't think it makes much sense to base important, irreversible decisions on fantastically extreme cases, especially when we could start with a simpler solution to which features (and concurrent complexity) can be added in the future, based on feedback informed by real-world experience.

If this is in reference to my example in https://github.com/tc39/proposal-pipeline-operator/issues/89#issuecomment-363639957, then I would like to push back on this characterization of it.

It is not the case that, simply because the example was contrived before a systematic review of real code was possible, therefore it does not illustrate problems that real code has to face. In particular, an asynchronous data transformation involving many other async functions may easily include many awaits in various positions in a nested/composed expression.

That expressions like:

`Response: ${
  await (
    await fetch(`https://example.org${path?.trim() ?? throw new TypeError()}`)
  ).text()
}\n\n`

…would occur in real code and that they would benefit greatly from left-to-right pipelining, such as (assuming a Proposal-2,3,4 pipe ■ placeholder):

path
  |> ■?.trim()
  |> ■ ?? throw new TypeError()
  |> `https://example.org${■}`
  |> await fetch(■)
  |> await ■.text()
  |> `Response: ${■}\n\n`

…or (assuming a Proposal-1 pipe whose RHS allows arrow functions rather than just bare identifiers):

await path
  |> $ => $?.trim()
  |> $ => $ ?? throw new TypeError()
  |> $ => `https://example.org${$}`
  |> fetch
  |> async $ => (await $).text()
  |> async $ => `Response: ${await $}\n\n`

…are not at all outside the realm of possibility. For that example, I just slightly elaborated an example from an article by Jake Archibald introducing async functions. This is perhaps less contrived than the example in https://github.com/tc39/proposal-pipeline-operator/issues/89#issuecomment-363639957, but it is not very different from it, and it demonstrates similar problems with the ergonomics of only allowing function calls in pipes.

But, in any case, this particular part of the debate is moot for now anyway. It will be moot until there is real data from a survey of real code, which I hope will eventually obtained, whether by me or someone before me, but which nobody has yet had time to do. It will be moot also as long as it has not yet been demonstrated that Proposal 4 could not possibly be implemented and understood in a simple fashion. We will try to implement the proposals in Babel and see what happens after that concretely. And hopefully we will be able to gather data about real-world composed data transformations involving non-function-call expressions. But before then it is premature to assume that complex non-function-call data transformations do not frequently occur in the real world. If I have been guilty of statements with similar assumptions, then I as well need to retract them, before any concrete demonstration either way.

Thanks for your patience in a long and convoluted discussion.

gilbert commented 6 years ago

I'm not certain that we need to nail down the specifics of that solution from the outset, rather than working towards it incrementally, and of the four proposals on offer, Hack style is my least favorite.

I want to keep the arguments fair. I know many have a strong preference for the F-sharp operator. And while that is a valid and significant factor in itself, it feels like some of its advocates are coming up with some unsubstantial arguments for it.

By itself the F-sharp operator works great for one use case and not much else. Although this is enough in languages like F-sharp, Elm, and OCaml, unfortunately it's not enough in JavaScript. It's not enough in Elixir either, or any language that doesn't have built-in currying. The language features, runtimes, and ecosystems just aren't designed around unary function pipelining. To introduce it "incrementally" wouldn't make sense when it has friction against the rest of the language.

In contrast, aside from the placeholder bikeshedding, the Hack-style operator seems to solve pretty much all the JS-related issues we've encountered so far. That's not to say it's perfect, doesn't need work, or is even the right proposal. But it works seamlessly with nearly all of JavaScript's existing features and libraries. It's for that reason Hack-style should strongly be considered.

kurtmilam commented 6 years ago

I want to keep the arguments fair.

This is my intention, as well. It's absolutely not my intention to argue unfairly for my preferred solution, and I don't think I have been guilty of that in this or any other discussion of the pipeline operator.

The two claims of mine that you seem to take issue with, summarized, are:

There exist tools (utility functions) that can be employed to partly mitigate the problem (mentioned by J.S. Choi) of increased verbosity that comes with using non-function-call expressions on the RHS of an F# style pipeline operator.
In the specific context of using a unary function inside a pipeline chain, Hack style will be more noisy than F# style.

Both of these are incontrovertibly factual statements. At no time have I ever claimed (nor was it my intention to give the impression) that F# style is less noisy than Hack style in all cases or even on average. I'm trying my best to stick to true and factual statements, to present fair arguments and to discuss the various proposals with honesty and an open mind. It's absolutely not my intention to use deception to try to tip the scales in favor of my preferred solution.

@gilbert I've followed you on Twitter. I'd be happy to continue this discussion in a respectful manner outside of this forum. I believe you are making a category error here, and I'd be happy to try to explain why I think that's the case, but I don't think this is the right place to do it.

js-choi commented 6 years ago

@kurtmilam: In order to minimize the chance of miscommunication, I’d like to ask whether the phrase “Hack style” in https://github.com/tc39/proposal-pipeline-operator/issues/89#issuecomment-364057444 specifically refers to Proposal 2: Hack Style Only, or all three proposals that involve Hack style at all (Proposals 2,3,4). While it is incontrovertibly true that unary function calls are noises in Proposal 2, it is not necessarily true in the Mix Proposals 3,4, and this issue’s scope covers the latter.

Also, please do link to the Twitter thread, if/when it occurs, back here. Thanks for both. 👍🏻

kurtmilam commented 6 years ago

@gilbert I'm referring to "Hack Style" only in that comment. I've stated elsewhere in this and other discussions that I support the Smart Mix and Split Mix proposals, although my current preference is for an F# solution (with room to grow in the future) in the operator's first iteration.

mAAdhaTTah commented 6 years ago

By itself the F-sharp operator works great for one use case and not much else.

I suspect there is significant overlap between expressions you can write with Hack-style pipes that would work the same if wrapped in an arrow function, and I don't think wrapping these with a familiar syntax is a heavy burden, although doing so is definitely noisier. Quite honestly, in the two examples @js-choi has above, the F#-style seems pretty reasonable, if less desirable.

That said, we should be able to bare this out with some real-world usage after we complete the babel plugins. I'm not a heavy user of async / await yet

...it feels like some of its advocates are coming up with some unsubstantial arguments for it.

This feels needlessly personal. Would you mind rewording?

@kurtmilam Advocating for F#-style and punting off the possibility of Hack-style pipeling to a future enhancement feels like shirking our responsibility to design this feature properly the first time. If you feel Hack-style pipelining is worth including, I think you should advocate for that now, rather than deferring that off to a future change / proposal.

gilbert commented 6 years ago

This is my intention, as well. It's absolutely not my intention to argue unfairly for my preferred solution

Let me say I never thought you or anyone has or had ill intention. Keeping things fair does not mean "looking out for bad guys", but just making sure things are played out properly. Just like a board game, someone might "cheat" by accidentally stepping over a rule if other players don't keep them in check.

The two claims of mine that you seem to take issue with, summarized, are...

These two claims have been qualified since their original introduction, and in their current form I agree with them.

gilbert commented 6 years ago

So I just discovered a new operator in JavaScript that works today! It's called $=

async function go (str) {
  let $;
  return (
    $= str,
    $= $ + $,
    $= $.toUpperCase(),
    $= Promise.resolve($),
    $= await $,
    $= log($),
    $= `oh ${$}`
  )
}
const log = (x) => { console.log(x); return x }

go("hi").then(log)
// "HIHI"
// "oh HIHI"

This is working code! We might not need the Hack-style operator after all 😛

rbuckton commented 6 years ago

So I just discovered a new operator in JavaScript that works today! It's called $=

Except that your $ variable is highly polymorphic, and thus might not be very efficient in an optimizing compiler.

js-choi commented 6 years ago

@littledan: I’ve signed the contributor agreement and started a spec repository for a Proposal 4. Other than IRC Freenode #tc39, and maybe the es-discuss mailing list, is there a more informal place I could ask for questions on Ecmarkup and for general guidance on the spec drafting? For instance, I’m currently having trouble getting Ecmarkdown to be recognized in an emu-intro element. But I don’t want to clutter up this issue with lots of questions. Thanks!

littledan commented 6 years ago

@js-choi Let's talk either on #tc39 or in issues here. Others might run into the same problems as you, so it's helpful to have an informal place.

I haven't tried using ecmarkdown in an emu-intro section before. I'm wondering, why are you putting grammar notation in an introduction? When I write introductions, I usually stick to plain text, maybe with some paragraphs, links, lists and code samples. I tend to put grammar in an emu-clause section. If you want to write an extended informal text to describe the approach to the solution, this could also be done in a separate Markdown explainer document.

As for not cluttering up this issue: Would it make sense for @js-choi and @mAAdhaTTah to each open up issues to track progress on their separate proposals? We can have more focused discussions there.

js-choi commented 6 years ago

@littledan: Talking on IRC #tc39; thanks for replying there. To answer your question above, I’m not trying to put Grammardown or other grammar notation in an emu-intro element: rather, I’m just trying to use Ecmarkdown for paragraphs. The introduction to Ecmarkup’s documentation suggests I should be able to use Ecmarkdown for “paragraphs”, at least, and I don’t see any documentation suggesting that emu-intro’s paragraphs are handled specially.

As for separate issues, I’m neutral, but it might be useful. Maybe it might be good to wait a bit until the specs and plugins are developed further. Once we continue developing the specs and plugins further, perhaps we’ll see how much discussion occurs around them. But I don’t know.

mAAdhaTTah commented 6 years ago

I'm starting with the babel plugin first, and I'll open an issue about it once it's ready.

mAAdhaTTah commented 6 years ago

Just a quick update, since things have quieted down here a bit: I stated working on |> await support, but since it doesn't currently parse (since await alone is not currently valid), it requires an update to babylon, which I'm still digging into.

js-choi commented 6 years ago

To add onto @mAAdhaTTah, us two have had professional work take up most of our time this past week, but we’ve also been able to put some work into the Babel plugins and formal specifications of our respective proposals (1: F-sharp Style Only and 4: Smart Mix) in our free time, while helping each other through email.

We have deliberately been working from opposite ends: @mAAdhaTTah has been working on Proposal 1’s Babel plugin first. I have been working on writing Proposal 4’s formal specification first. Many problems with Babel that @mAAdhaTTah will run into are problems that I would also run into—likewise, the problems I’d run into with the spec and with Ecmarkup would also apply to both of us.

So we figured it’d be most efficient to start work from the two opposite ends (plugin and specification), then adapt each other’s work for the half we haven’t worked on.

My own specification’s repository is public, as I mentioned in https://github.com/tc39/proposal-pipeline-operator/issues/89#issuecomment-364661590. Anyone is free to watch its progress, but it is still quite unfinished and rapidly churning.

I’ve mostly figured out how Ecmarkup works, but I have been churning out an explainer and a specification together in the Markdown readme first; the Ecmarkup document will come later. I hope to have my first presentable and well-rounded explainer+specification by the end of this week, but the planning fallacy will probably strike me, so it’ll hopefully only be a little bit later…

We’ll keep you posted later this week.

js-choi commented 6 years ago

An update after a week: We’re still working on the problem from opposite ends.

The Babel plugin for Proposal 1 (F-sharp Only with … |> await |> …) has run into some parser complications. More specifically, Babylon the parser itself would have to be patched to handle any bare await lacking a following identifier (see https://github.com/tc39/proposal-pipeline-operator/pull/85#issuecomment-364699995). @mAAdhaTTah is still working hard on it and we will hopefully have something to show in this space soon.

A first-draft formal specification for Proposal 4 (Smart Mix) has been finished and uploaded to my personal website. The draft is barebones but I think it completely specifies all the basics of Proposal 4 (Smart Mix).

In contrast, Proposal 4’s explainer is not yet ready; the explainer will take a few more days before it is presentable for first review. I want to completely remove the current middle section, which is both now out-of-date and redundant with the formal specification. I expect that many questions regarding the specification’s design will be answered by the finished explainer.

I also want to add more real-life examples from real-life libraries to the explainer. Right now it uses code from/with Underscore, Pify, and the WHATWG Streams standards, but I also want to add examples from/with Lodash, jQuery, Flux, Cycle, RxJS, and Ramda. If you have any other suggestions for important JavaScript libraries with which to show demonstrations (both from within their implementations or from code using their public APIs), please let me know.

I also still want to add some more brief informative comments to the specification and maybe even add tentative grammars for add-on proposals, like pipe functions (see issues #96 and #97), in an appendix of the specification.

After rewriting the explainer and adding more real-life examples, I plan to then either move onto Proposal 4’s Babel plugin, start a formal specification for Proposal 1, or start a formal specification for Proposal 3 (Split Mix).

I’ll leave another update when the situation changes further or after another week – whichever comes earliest. Thanks.

js-choi commented 6 years ago

Another update:

@mAAdhaTTah and I are still collaborating, mostly via email. He's been working hard on extending Babylon (the raw parser of Babel) to allow |> await. He has submitted a Babylon pull request at babel/babel#7458 with his work, which recently got approved from a reviewer. We’ve been having some conversations there about how to group the proposals in the Babel plugins (one plugin with configs or separate plugins).

We’ve both started to work on implementing the Babel plugin in the same repository.

I myself have finally finished a detailed explainer and specification for Proposal 4: Smart Pipelines, plus several possible extensions to it. @littledan will be able to look at it in a few days, and if he has time he will prepare a presentation presenting it and Proposal 1: F-sharp Style Only to the next TC39 meeting, later this month.

100 (feel free to read and give feedback)
Readme explainer
Formal spec

Thanks for all your patience.

tc39 / proposal-pipeline-operator

Discussion: Proposal Mixes #89

100 (feel free to read and give feedback)