Introduced Observable >> operator as pipe shorthand

[jackgene]

I was doing something with F# and ReactiveX, and noticed that because of F#'s |> operator, their pipelining syntax is a lot cleaner and more readable:

    let reqWithTimeout =
        req
        |> Observable.timeoutSpan (TimeSpan.FromSeconds 1)
        |> Observable.take 1
        |> Observable.catch Observable.empty

Here's the full F# ReactiveX Application.

So I thought that by overloading Python's right shift operator (__rshift__, >>), we could get something similar:

        req() >> ops.timeout(datetime.timedelta(seconds=1)) >> ops.catch(rx.empty()),

Here's the PR converting the entire RxPY application from pipes to >>s.

While overloading the >> operator to perform pipelining seems like a strange choice, it is not without precedent, as Airflow is doing the same thing (note the first_task >> second_task >> [third_task, fourth_task] example).

WDYT?

[dbrattli]

This becomes too much "magic" imo.

1. This is not the way things are normally done in Python
2. The operator >> means compose in F#, not pipe
3. Using | would perhaps be better but still too much "magic"

I would be more in favor of re-adding method-chaining syntax similar to C# 🙈 as an alternative to pipe. This is what I have done in the Expression library which supports both pipe and method chaining.

[jackgene]

This becomes too much "magic" imo.

1. This is not the way things are normally done in Python
2. The operator >> means compose in F#, not pipe
3. Using | would perhaps be better but still too much "magic"

I would be more in favor of re-adding method-chaining syntax similar to C# 🙈 as an alternative to pipe. This is what I have done in the Expression library which supports both pipe and method chaining.

~Re-introducing method chaining makes sense as well.~

~To be completely honest, the rationale for moving to pipes was never clear to me, and struck me as a step backwards. I thought it was just one of those things that made implementing RxPY easier.~

I had a closer look at pipe(), and along with the discussion here, I think I better understand its rationale. I was initially just looking at code examples, and what writing applications look like from an object-oriented perspective.

But it looks like it's more about moving to a function-base approach, and an operator is simply a function from one observable to another. The introduction of pipe() it to make it possible to chain operators (instead of nesting them).

So with that understanding, I think in the interest of:

There should be one-- and preferably only one --obvious way to do it.

I'm not so sure about re-introducing method-chaining.

[MainRo]

Moving to pipes allowed implementing operators outside of the rxpy library. This is invaluable to me. I always felt that operator overloading was awkward. This is used in many different languages but it always ends up being strange, and and non-standard related to these languages best-practices.

For several years, I have been thinking about writing a pep similar to the javascript proposal for a |> operator: https://github.com/tc39/proposal-pipeline-operator

This js proposal has never been accepted, but most of the proposal applies to Python. What do you think about going this way as the next big step ? I would be interested in co-writing such a pep if there is some interest.

[dbrattli]

Yes, I have been thinking about a pipe operator myself for some time. The F# pipe operator however have some problems with Python since we don't have currying and writing nested functions like we do is a bit awkward. My curry_flip decorator helps, but it can be very confusing. The hack pipe proposal looks very interesting, let me have a look ...

[jackgene]

Introducing a dedicated pipe-operator in Python seems like an interesting idea.

I'd be interested in chipping in, for now, I just want to share some off the cuff thoughts.

IMO, the crux of the problem isn't because function currying isn't Pythonic (and rarely used) like it is in ML-based language (such as F#), but rather a problem with partially applying functions.

Let me elaborate on what I mean. First of all, I think piping will work just fine with single-argument functions, so I don't think we need to go too deep into it. Using |> as an example (since we all seem familiar with it):

import math
import rand

math.sqrt(abs(random.randint(-1000, 1000)))

# Equivalent to:
random.randint(-1000, 1000) |> abs |> math.sqrt

Note that abs, and math.sqrt are both single-argument functions.

The problem becomes more interesting when dealing with function that takes more than one argument and needs to be partially applied:

from functools import partial, reduce

reduce(
    lambda l, r: l + r,
    map(
        lambda n: n * 2,
        [1, 2, 3, 4, 5]
    )
)

# Equivalent to:
[1, 2, 3, 4, 5] \
|> partial(map, lambda n: n * 2) \
|> partial(reduce, lambda l, r: l + r)

Or worse, when you need to skip applied arguments (we need to apply the first and third argument to reduce, skipping the second):

from functools import partial, reduce

reduce(
    lambda l, r: l + r,
    map(
        lambda n: n * 2,
        [1, 2, 3, 4, 5]
    ),
    0
)

# Equivalent to:
[1, 2, 3, 4, 5] \
|> partial(map, lambda n: n * 2) \
|> lambda nums: reduce(lambda l, r: l + r, nums, 0) # functools.partial does not work

So on the surface of it, I see this as requiring two PEPs:

1. Introduce the pipe-operator (called |>, or perhaps something more Pythonic, Kotlin uses let, for instance). I feel like this would be the simpler proposal.
2. Introduce language-level support for partially applying functions. I can think of two issues with functools.partial that needs addressing (though there likely will be others):
   1. The syntax is somewhat verbose, so language-level support would make it more succinct (Hack's approach seems promising).
   2. It currently only partially applies parameters from the left (so in my example above, you can partially apply reduce's function=... argument, but not initial=...). You could argue that ML-based languages that rely on currying as its basis for partially applying functions have the same problem, but keep in mind that ML libraries tend to be designed with the pipe-operator in mind, and positions arguments accordingly (hence F#'s Sequence.fold (the equivalent to functools.reduce in Python) has its state (Python reduce's initial) as its second argument). Python already has a rich ecosystem of libraries that may not necessarily have been designed with piping in mind.