[RFC] Pipe Operator

[felixbuenemann]

It would be great if Crystal had a construct similar to Elixir's pipe operator |>:

class Importer
  def import(io)
    xml = parse_xml(io)
    root = find_root_node(xml)
    tree = build_tree(root)
  end

  def import2(io)
    io |> parse_xml |> find_root_node |> build_tree
  end
end

I think that the latter example is much easier to read and requires far less eye tracking to comprehend.

The implementation in Elixir is simply syntactic sugar using a macro, not sure if Crystal's macros could do this transform as well.

This has been briefly discussed in #1099, but I don't think the arguments against it were valid. Crystal is just like Ruby a hybrid between an OO and functional language and I really like the funtional core, imperative shell pattern, which leads to well testable units by avoiding shared/hidden state.

[jhass]

We're fairly thin on the available syntax space already, so we should be very careful in using that up.

I'm not sure I find it more clear personally, it's something that makes sense because you're used to it, but you have to learn it. Syntax is always something to learn, I hang out on #ruby a lot, and "What does &: do" is one of the most common questions, people don't even recognize that it's &(:foo) not &:(foo). The point is not about drawing an analogous ambiguity in your proposal, but why I think that too much syntax can be harmful in learning a language.

[felixbuenemann]

People already now how to use pipes from the command line, so I think it's pretty easy to grasp.

[jhass]

Or are they? On the command line the pipe passes the output to an invisible parameter (stdin) that doesn't even exist in Crystal, here it magically turns a argument-less call into one with an argument (now think about the ambiguity that adds if there's an overload that takes one and one that doesn't). The analogy would rather be "a |> b looks just a | xargs b", which it does not.

[asterite]

This was briefly discussed in point 4 of #1099.

Right in your first example you have:

xml = parse_xml(io)
root = find_root_node(xml)

In Crystal, right now, you can do this:

require "xml"

xml = XML.parse("<foo>1</foo>")
xml.root

That is, you invoke a method on the object. The "implicit argument" of Elixir's |> is an explicit thing (the call receiver) in Crystal and other OOP languages. Elixir does that because it's mainly a functional language, but Crystal is mainly an OOP language.

In the main example in Elixir's pipe operator I see:

[1, [2], 3] |> List.flatten |> Enum.map(fn x -> x * 2 end) #=> [2, 4, 6]

In Crystal you can do it like this:

[1, [2], 3].flatten.map { |x| x * 2 } #=> [2, 4, 6]

There's also the thing, that I kind of dislike, that the pipe operator assumes the result will go as a first argument in the function. That means that when you define a function you have be aware that the first argument might be used with a pipe operator, which forces you to stop and think for a moment. In Crystal this is no real issue because the call receiver is that piped object.

For the cases where you do need to pipe things as method calls, I wouldn't mind having explicit variables and arguments. This shouldn't be that frequent, or bothersome.

I wouldn't try to introduce mainly functional features into the language. There's also what @jhass says: more syntax means everyone will have to learn it.

[felixbuenemann]

@jhass The command line equivalent of the above code would be:

parse_xml < io > xml
find_root_node < xml > root
build_tree < root > tree
# vs.
cat io | parse_xml | find_root_node | build_tree

So I think they are very much equivalent.

This syntax doesn't cover all use cases, but it is very well suited to cover the common transformation pipelines where you have several intermediate representations before arriving at the final result.

There are of course OO approaches to handling this, eg. think about traversing the dom hierarchy tree, like @asterite mentioned. These are beautiful to use as a public API, but they are IMHO overkill for simple cases because they require shared state and much more code to implement than the simple data in, data out approach, which is fine for a large part of transformations.

@asterite My main intention in the example was to illustrate the case were there are several independent function that taker one piece of data in a certain type and transform it into another peice of data of a different type. Sure, there is an XML parser with a dot-chaining API, but writing an endless chain of xpath selectors and enumerators will get at the right data, but will be undecipherable to someone else or me in two weeks. Breaking these complex steps into smaller composable functions makes the code much easier to understand, because I can just look at the function names and the order they are called to understand what's going on. The pipe operator makes this very common use case much more readable.

As you pointed out, it does not make sense to use this with enumerators, because these are already chainable, but this doesn't make the other use cases less relevant.

One thing I really like about the pipe syntax, is that is removes all the noise of the intermediate variables without having to write any extra glue code.

[beno]

+1 for adding a pipe

It is a well known pattern from Elixir, F# and Haskell.

Take a look at this talk for how it can greatly improve code readability.

Bonus points: Crystal can be billed as a functional language when we add this!

[waterlink]

Have yourself some pipe: http://carc.in/#/r/fg2 ;)

[waj]

Oh please, don't do this to the language :-P

Adding too many features is bad. That's why we take so long before adding any simple thing. But using macros to imitate those features is even worse. Macros should be used to avoid boilerplate, to do metaprogramming, but not to invent new language features. Otherwise this could escalate too quickly to the point where it's impossible to follow others source code.

[waterlink]

Agree to @waj

I think macros here just gives you raw power, and it is your responsibility to use it correctly ;)

[beno]

I am all for a bit of philosophical debate on How Things Should Be now and then, but dismissal of a demonstrably useful feature based on some generic observations of what is Bad and/or Good is... rather disappointing. Oh well.

[asterite]

@felixbuenemann @beno You already showed an example where pipes could be useful and I showed how to do it with Crystal in a way that doesn't need pipes and it's even shorter and more understandable.

If you can show us some real Ruby/Crystal code (you can search it on the internet, or maybe some code that you wrote, or some real (not made up) code that you want to write) and later show us a rewrite with pipes, we can judge if it's a valuable addition to the language.

It's not that we don't like pipes, or based on a few observations we dismissed it. It's just that when you design a language you have to be careful not to add too many things to it, specially redundant things (because everything you can do with pipes you can do without them).

Also, the sample code in the slides for Swift could be rewritten to do: return false unless ... and then you get a flattened method that's pretty easy to understand.

[beno]

Thank you for your response.

because everything you can do with pipes you can do without them

I don't think that's a good argument. You may as well drop the multiplication operator because you can just add a whole bunch of times :)

Pipes have a great story to tell wrt readability, composition and reuse, much more so than blocks. Yes you can chain enumerators like you showed, but being restricted to the methods that the return value has is very different from being able to mix and match any function that accepts certain inputs.

Anyway, hopefully you'll give it some more thought, watch a few videos and play around with some of the languages mentioned.

[asterite]

The main problem is that in OOP you don't use free functions as much as in functional languages. You do:

[1, 2, 3].map { ... }.select { ... }.sort

instead of:

Enumerable.sort(Enumerable.select(Enumerable.map([1, 2, 3]) { ... }) { ... })

which could be rewritten with the pipe operator to:

[1, 2, 3] |> Enumerable.map { ... } |> Enumerable.select { ... } |> Enumerable.sort

Also note how repetitive the code becomes: do we really need to specify that we are working with an Enumerable each time? It's obvious from the object type.

That's why I'm saying that this feature is very useful in function language because of the way you structure code. In OOP you don't structure code like that, there are not many free functions and so the operator becomes much less useful.

[felixbuenemann]

You miss the point that pipes are not a replacement for enumerables, but a nicer syntax to compose stable functions.

Despite that the example given above could be written as:

[1, 2, 3] |> map { … } |> select { … } |> sort

As long as Enumerable is included into the current context.

However in this case I wouldn't use the pipe op, because I could just as well use chaining, because the Enumerable methods above all return an enumerable. My argument for using the pipe operator was to combine functions that return different types that cannot be chained or would require lots of boilerplate to be chained.

[beno]

I'll be blunt: I don't see a big future for Crystal if it insists on staying fully OO and only OO. OOP is past it's peak and there is a big trend towards more functional and hybrid OO/functional styles in a lot of modern languages due to the inherit benefits FP (and also immutability, but that's another story) has. I really wish Crystal would recognize and embrace that trend instead of resisting it. But you need a decent level of understanding before you can embrace it, and based on the above that just isn't there right now. So I reiterate my encouragement to keep learning. Please don't be offended by this, I have the utmost respect and appreciation for all the work everyone is doing on Crystal and am fascinated by the language. It is precisely why I hope it won't stay purely OO.

[bcardiff]

@beno , @felixbuenemann I encourage, support and love functional programing. A lot. I like crystal to be as flexible as it can (not everybody agrees with this though ;-) ). Not everything is achievable by macros and it makes difficult to share code. But you can do plenty of stuff for syntax sugar things.

The language is young. With lots of things to offer. Lots of good people's energy is involved.

In the hybrid space of OO/λ there is plenty of space to draw a line. Huge chances of not ending in the one everybody would like :-). But me and probably lots of people that are investing energy think there is future despite the place of that cut line.

[refi64]

OK; I'm still confused. I mean, I love FP...but what does it in particular have to do with the pipe operator? I don't think Haskell even supports a left-to-right data operator. Most commonly, you'll see something like this:

myMagicFunction x = a . b . c $ d x

(I know about currying, but that distracts from the point here.) Crystal can already do that:

def myMagicFunction(x)
    a b c d x
end

Left-to-right vs. right-to-left doesn't have much to do with FP in particular. It's mostly about higher-order functions, composition, and immutability (this one's frequently argued). Not...an operator?

(Of course, maybe I missed another post here. :)

[bcardiff]

@kirbyfan64 I love how lots of features just blend together in Haskell for example, from conventions of ( ) in types vs in expressions, together with curry and lazy evaluation, later to bind, monads and procedural-like code lazy evaluated. The strong type inference and how neat is to deal with infinite structures. Build your dsl.

We are trying a new language, that borrows a lot of others. It won't work to mix all the ingredients of all the languages all at once. Like food.

Something I didn't stand before is that I like the idea of a pipe operator. Mainly because is sugar, is little, I can get used to. But I am not sure is such a good idea, at least, right now. I haven't yearned (yet?) for those in industry.

And last but not least, I like multi-paradigm languages. If someday I am able to mix Prolog like, with λ and OO in a nice way... oh my! :-)

[felixbuenemann]

I still believe that the pipe operator (or whatever you want to name it) would be a great addition to Crystal's syntax without feeling out of place. It it IMHO much easier to read than either right-to-left (inside->out) nesting of function calls or assignment to intermediate vars.

Noting that for some use cases this could be replaced with method chaining is not a good argument against it, because that's not the problem it would solve in Crystal.

Crystal is in big parts inspired by Ruby which features a very human readable syntax compared to other programming languages. This feature would make function composition a lot more human readable, which is why I think it would be a great addition to the language.

[js-ojus]

@felixbuenemann : You say "My argument for using the pipe operator was to combine functions that return different types that cannot be chained or would require lots of boilerplate to be chained."

Could you please illustrate that with an example? Thanks.

[felixbuenemann]

@js-ojus It seems my initial example was a bit shallow with all the methods omitted, I'll try to flesh it out this weekend.

[bjmllr]

At a recent conference, Matz gave a talk about Ruby 3.0 that included a description of how a pipe operator superficially resembling Elixir's might be introduced as a form of concurrency.

There's also an ongoing discussion of function composition operators in this Ruby feature request.

[js-ojus]

A few observations.

• Normal chaining of free functions reads backwards in most FP languages. I have always felt that the pipe operator is the FP world's attempt at emulating method chaining of OOP world. Introducing it in an OOP language sounds like looking through a mirror in a mirror, when the original object is directly visible!
• Having said that, I refer to a recent mailing list conversation that I had with @asterite and @waj. The topic was higher-order messaging (HOM). Higher-order messages are OOP version of composed free functions.
• It may not be possible to get straight function composition a la Ruby's proposal cited above, because of Crystal's static typing. @asterite ?
• Not withstanding, please see Scala's Product family, Tuple family and Function family for an example of what full composition has lead to in a statically-typed, inheritance-oriented language. Note how Scala predefines products and tuples up to an arity of 22!
• Fully-general composition in a statically-typed, inheritance-oriented language is inevitably linked to variance, as well.

[ozra]

Far better than pipe-notation would be UFCS. Then you use the same familiar syntax for chaining calls and using previous return value as first param for next func. You get the same left-to-right ordering benefit, without straying from syntactical style. I've coded some with pipes, but definitely prefer the 'dot-notation' over it - less noise. Proposition: when method-syntax: first look for methods on type, then for "functions" with type as first arg. When using func-syntax: the other way around. I'm still reading discussions on the thoughts on pros and cons of different ways of tackling the nuances of it.

[refi64]

@ozra Two things:

1. You could have bothered to explain what UCFS is! ;)
2. I don't like using dot syntax to call unary functions in OO languages. I personally think it makes code harder to read, especially in languages like Crystal, where you can extend a class to add more methods later on.

[ozra]

Thanks @kirbyfan64, fast elaboration: UFCS means Uniform Function Call Syntax. You can call some-method arg and arg.some-method interchangeably. For Crystal this means that you can declare a function that matches a range of types as first argument, and it will be a secondary choice for overloading for all those types whenever a method is used on them that match the name. And vice versa - if wanted to support free-func style for methods. The upsides:

• One single def for several types, when this is appropriate (I have some cases where it really makes sense imo)
• While you could have re-opened a class - the function still just has method-access, no direct member-data access, which decouples it further.
• The already established re-opening scheme is what makes UFCS a natural extension imo.
• No ugly pipes ;-)

[stugol]

+1 for UFCS (or pipe syntax).

[asterite]

I'm closing this. This syntax is just a big of sugar that will mean all users will have to learn it while right now using a local variable and passing that explicitly works and can be understood by everyone, and has the same performance. Let's keep the language simple.

[kristianmandrup]

I understand that the Crystal Corey team want to keep it typical OO and Rubyish. However anyone coming from a functional mindset would love to see built-in support from currying and pipes, like in OCaml, F# and that ilk.

I guess the only way in that direction would be to fork it and make a functionalists first variant of Crystal. Otherwise better to stick with Ocaml, F# etc.

[tibastral]

Pipes are more than sugar.

in elm we have :

add a b =
  a + b

and then we can have

add2 =
  add 2
add3 =
  add 3

add2Plus3To3var1 =
  add2 (add3 3)

add2Plus3To3var2 =
  (add2 << add3) 3

add2Plus3To3var3 =
  3 |> (add2 << add3)

and we can think in terms of function composition and flux of data transformation and not in terms of fonction arguments

I know that it would look less rubyish though

[kristianmandrup]

Also see: https://github.com/hopsoft/pipe_envy - Pipe operator for ruby

Videos https://www.youtube.com/watch?v=ThB2cpPsb1o - Pipe operator for ruby

"Elixir is one modern language that is introducing many Rubyists to the world of highly scalable, highly distributed, functional programming-based programming. In a more narrow scope, one language feature that many people liked is the now famous Pipe Operator "|>". There is nothing like this in Ruby. But could there be such an operator? And if it could be done, would it be useful? I started a pet project called "Chainable Methods" to address just that. "

[kristianmandrup]

I'm working on a project to make any language extensible, while still working with the same underlying parser/compiler infrastructure with no intrusion needed.

Will work by extending existing infrastructure, using source maps to compile special constructs (such as pipe operator) to target language and sending that for evaluation in the LSP (Language Server) so that you get all the usual IDE/editor benefits.

[straight-shoota]

@kristianmandrup That project sounds really interesting. But I fear that this will only lead to more segmentation because the code depends not only on the compiler but on an intermediate. This means effectively having two versions of the same language. In my opinion, that's the worst solution. Both, having the feature directly in the language or not using it at all seem to be better.

[kristianmandrup]

@straight-shooter What I intend with this model is to make it feasible to have many higher level variants of any language. It would still compile to equivalent code of target language, 95% of the code one to one, untouched.

Would allow for any developer or organization to quickly configure it for their own preference to be more productive in their "style". Think of it more like choosing a color theme of your editor or adding extra tooling. Anyways, not pipe or crystal specific. I think this could be a major revolution, much like BabelJS was for JS.

[ShalokShalom]

Understanding what pipes do, costs you about 10 minutes and then saves you from stress for the rest of your programming career.

So 'it takes time to learn' is counter-intuitive.

Can you do that in Crystal?

With the same LOC and the same clarity and safety?

Completely staggering to me, how people with so little insight into the language that originated pipes, decide on something so important to be included in the language.

FSharp is as much an imperative language, as an object-oriented and functional language.

It is a fully functional super-set of C#.

Logic is by definition side-effect-free and core part of all kinds of applications, the argument that Scala messed it up to combine composition with its imperative part and that is, why we should avoid Pipes in Crystal is kinda ridiculous.

Scala 3, F#, Kotlin, Rust, and half a million other languages show, how it can be done.

If you are truly interested in the topic, read 'domain modeling made functional' and no, you don't need a super sophisticated functional language to do that.

[straight-shoota]

The link you posted doesn't work.

[ShalokShalom]

Thanks, I corrected it.

[asterite]

The link points to a comment. So the answer is yes: Crystal can do comments to.

Just kidding. What specifically from that you can't do in Crystal? Because Crystal is Turing complete

[ShalokShalom]

With the same LOC and the same clarity and safety?

[asterite]

I don't think so. Crystal would probably be more concise. Or that's generally true for Crystal.

Maybe you could explain what that code does and we can all translate it to Crystal?

[ShalokShalom]

You don't even know, what it does, and you know Crystal is more concise?

[asterite]

I program daily in Haskell, and I've programmed in other functional languages. In my experience, Crystal is more concise.

[ShalokShalom]

So then you know what this code does.

[asterite]

I didn't take a look at it... Is it a parser combinator?

[ShalokShalom]

Yep

[asterite]

Check this out: https://github.com/dhruvrajvanshi/crystal-parsec

[ShalokShalom]

That's nice. CRZ is sadly not supported anymore, and this package has also gotten no new commit for almost 3 years.

I don't try to diminish the language, it simply seems, that it does lack some core integral tools, that are important for me, and there seems to be no love for functional programming.

I also don't think, that experience in Haskell can make up for experience in FSharp and its style of programming. Same as FSharp cant replace Haskell type classes, despite both languages being similar in other cases.

And if Crystal can do all of this - and even shorter - then is the documentation lacking behind the language.

[straight-shoota]

if Crystal can do all of this

What does "all of this" refer to? You still haven't properly explained what exactly you're talking about. The previous comments are all not really helpful for that.

[ShalokShalom]

Dot chaining comes with all the typical boundaries of oo programming, like being dependent on classes. I like flexibility and independence. I just agree with all the others before me, who voted for pipes: Functional programming has a reason and is imho too less supported in Crystal.

And 'all of this' refers to the code below the point, I referred too, that's why it points to its header comment.

[straight-shoota]

And 'all of this' refers to the code below the point, I referred too, that's why it points to its header comment.

To really comprehend what this means, it would be helpful to show a ideally small example of code with pipe operator, explain what it does and what the equivalent implementation with current Crystal looks like. This was done in the OP and numerous other comments on this and related topics. Without explicit code, it's impossible to reason about drawbacks and benefits. Especially for people not familiar with fsharp.