backcalls: let's add them

[michaelficarra]

Copied proposal from https://github.com/jashkenas/coffee-script/issues/2662#issuecomment-13026081 below

---

Taken pretty much from the coffee-of-my-dreams list, the best way to explain my preferred backcall syntax is through example:

• The continuation is captured and tacked on as the final arg by default

  <- fn a
  ...
  b
  

  compiles to

  fn(a, function(){
    ...;
    return b;
  });
  

• The left side of the arrow specifies the argument list for the continuation. When only one is given, parens can be omitted.

  a <- fn b
  ...
  c
  

  compiles to

  fn(b, function(a){
    ...;
    return c;
  });
  

• And for the rare occasion where the continuation is not the final argument of the function, we need to be able to specify where to put it with a marker like the <&> I use below. I'm open to suggestions for a better marker we can simply use an anonymous function to force the position of the callback.

  (a, b) <- (_) -> fn c, _, d
  ...
  e
  

  compiles to

  (function(_){ return fn(c, _, d); })(function(a, b){
    ...;
    return e;
  })
  

  or, if we detect this case, the simpler:

  fn(c, function(a, b){
    ...;
    return e;
  }, d);
  

this must be preserved in the continuation by rewriting references as we currently do in bound functions. I don't think we should worry about arguments rewriting, but that's your call.

Here's a real-world example @paulmillr gave in #1942:

compile = (files, callback) ->
  async.sortBy files, @sort, (error, sorted) =>
    return @logError error if error?
    async.map sorted, @map, (error, mapped) =>
      return @logError error if error?
      async.reduce mapped, null, @reduce, (error, reduced) =>
        return @logError error if error?
        @write reduced, (error) =>
          return @logError error if error?
          do @log
          callback? @getClassName()

and here it is using backcalls:

compile = (files, callback) ->
  (error, sorted) <- async.sortBy files, @sort
  return @logError error if error?
  (error, mapped) <- async.map sorted, @map
  return @logError error if error?
  (error, reduced) <- async.reduce mapped, null, @reduce
  return @logError error if error?
  error <- @write reduced
  return @logError error if error?
  do @log
  callback? @getClassName()

edit: dropped the callback position indicator, opting for "hugs" style

[00dani]

As observed back on #2662, backcall syntax works well with RequireJS:

($, someLib) <- require ["jquery", "someLib"]
codeThatUses $, someLib

Another potential use is amb:

x <- amb [1,2,3]
y <- amb [4,5,6]
amb() if x + y < 10
x * y

[michaelficarra]

Also, for any Haskell fans, yes, this is just do notation under a different name. But don't let the procedural programmers know, or they will reject it as "elitist".

[summivox]

And for the rare occasion where the continuation is not the final argument of the function, we need to be able to specify where to put it with a marker like the <&> I use below. I'm open to suggestions for a better marker.

From the previous post I proposed ^ (caret) as non-final callback placeholder. Justification: visual hint of control flow.

(a, b) <- fn c, ^, d
# use (a, b)
e

[satyr]

JFYI, it originates from #1032.

[michaelficarra]

Also, we could always drop the concept of a placeholder and force people to force the callback as the final argument. There's @00Davo's approach from #2662 of defining higher-order functions such as flip:

flip = (f) -> (x, y, rest...) -> f y, x, rest...
<- (flip setTimeout) 250
do takeAction

And there's also just manual partial application:

<- (_) -> setTimeout _, 250
do takeAction

[paulmillr]

i’d vote for dropping placeholder +1 it brings more complexity, still

[epidemian]

And there's also just manual partial application:

<- (_) -> setTimeout _, 250
do takeAction

Using the right variable name you can have a person giving you a hug there:

<- (u_u) ->

---

I'd also vote against having the placeholder syntax; unless that concept of placeholder could be extended and used elsewhere too. In general i'm not much of a fan of this proposal, but i must admit that the use cases @00Davo mentions are quite really :smiley:

[shesek]

Is there any reason to automatically return inside the callback? its most likely that nothing would consume the return value of the callback anyway.

[ghost]

I'll throw in my hat in favor of the placeholder syntax. Just reading the code, this:

(a, b) <- fn c, ^, d
# ...
e

is infinitely more clear than this:

compose = (f) -> (c, d, callback) -> f c, callback, d
<- (compose fn) c, d
# ...
e

The latter also requires a separate "helper" function for each non-"standard" argument style that needs to be adapted for, harming interoperability with existing code.

[vendethiel]

A compose function is unneeded here. I'm in favor of the hug approach

[ghost]

@Nami-Doc Ah, missed that. Yep, that looks good to me.

[epidemian]

Is there any reason to automatically return inside the callback? its most likely that nothing would consume the return value of the callback anyway.

Why not? In one of the use-cases mentioned in this thread, defining AMD modules, returning a value from the backcall would be really important:

($, _) <- define 'thing', ['jquery', 'underscore']
# Very typical use-case for AMD modules: returning a constructor function.
class Thing
  foo: -> # ...

[ghost]

What if intermediate callbacks need to have a value returned? Would backcalls not be usable in this situation?

[vendethiel]

Why not?

Because we can't no-op them if chained (for example).

<- epi
<- dem i for i in [0..2]
an

I'd probably vote +1 tho.

[ghost]

@Nami-Doc I don't understand. Couldn't you tack a return, undefined, or null to the end of that snippet if you don't want it doing an implicit return?

[vendethiel]

but that would no-op the inner backcall, not the one with the loop ;-).

[michaelficarra]

@Nami-Doc: That's why you would never write it like that.

<- epi
for i in [0..2]
  <- dem i 
  an
return

or my preferred

<- epi
for i in [0..2]
  dem i, -> an
return

I hate bad code strawmen. You can write bad code in any language, we get it.

[ghost]

@Nami-Doc Ah, so

<- dem i for i in [0..2]

translates to

for i in [0..2]
  <- dem i
    # ...

and not

<- dem (i for in in [0..2])
# ...

?

[vendethiel]

Considering dem i for i in [0..2] is (dem i) for i in [0..2] probably - coco treats it as an invalid callee (array)

[ghost]

Thinking about this a bit more, conditionals and loops complicate things quite a bit. Consider:

if a
  b = <- fn c
  d = b
else
  d = e()

console.log d

b =
  if a
    d <- fn c
  else
    d <- fn c

console.log b

console.log
  for item in array
    element <- transform item

result = null

console.log
  until result?
    result <- generate

Lots of messy edge cases to handle.

[vendethiel]

Why are they complicated in your examples?

[michaelficarra]

@mintplant: They don't complicate anything at all. The captured continuation is the rest of the block. Maybe you're thinking it's the rest of the function/program?

[ghost]

Ah, I see what you mean. I suppose I misunderstood the scope of this syntax addition. So, these only help with non-branching code paths, then?

[epidemian]

dem i, -> an

WTF happened to my name? xD

Anyway, between all those messages i got lost. The consensus was that it does make sense to return values from backcalls, wasn't it?

[michaelficarra]

@mintplant: They work perfectly fine with branching code paths. It is a very simple transformation, you're overcomplicating it.

@epidemian: Yeah, it makes sense to auto-return from backcalls. That's why satyr/coco does it.

[vendethiel]

WTF happened to my name? xD

Sorry, no idea why I came up with that :p . And yeah, that'd make sense.

[ghost]

@michaelficarra Right, I get that now. I never said they didn't work with branching code paths, just that they don't provide any special functionality to help account for them, which I had hoped they would, but now see is outside the scope of this change. Sorry for the misunderstanding.

[00dani]

The "hug-style" <- (u_u) -> is amazing.

As for the higher-order function approaches, this seems problematic:

compose = (f) -> (c, d, callback) -> f c, callback, d

Because it looks nothing like a standard definition of compose and has totally different semantics.

I think very simple higher-order manipulations, like flip, are reasonable to use, though, unless they have a big performance impact at runtime (although I doubt they'd really have a big impact?).

[michaelficarra]

Hugs it is! I will edit the original proposal to omit the callback position indicator.

edit: done. Copied below:

---

And for the rare occasion where the continuation is not the final argument of the function, we need to be able to specify where to put it with a marker like the <&> I use below. I'm open to suggestions for a better marker we can simply use an anonymous function to force the position of the callback.

(a, b) <- (_) -> fn c, _, d
...
e

compiles to

(function(_){ return fn(c, _, d); })(function(a, b){
  ...;
  return e;
})

or, if we detect this case, the simpler:

fn(c, function(a, b){
  ...;
  return e;
}, d);

[vendethiel]

This would make @tenderlove happy

[summivox]

Although the "hug" admittedly does the job, it's more or less desugared, adds syntactic noise, as well as introducing a layer of unnecessary concept hurdle to learners. By inventing a new syntax, we're already trying to contract an idiom. Makes no sense to me to introduce a new one in the process.

Is there a convincing reason to hate placeholder?

[shesek]

As a more generic solution for passing the callback in an arbitrary position, one could use something along the lines of:

marker = {}
midcb = (fn, args..., cb) -> fn.apply this, args.map (arg) -> if arg is marker then cb else arg
<- midcb bar, 1, 2, marker, 3, 4 

[vendethiel]

"But duude, we could get a hug operator!" wasn't partial application refused already?

[michaelficarra]

@smilekzs: We're trying to make the language "simpler", or in other words, have fewer concepts. If we can combine backcalls and lambdas to support the rare case where the callback is not the final parameter, it allows us to have a simpler language than if we were to add a special placeholder syntax for that extraordinary case. That's why everyone was so in favour of using existing constructs.

[summivox]

I totally understand the call for simplicity.

I instead see the placeholder as an integral part of the backcall syntax, not an added burden. It's intuitive to look at the "caret" as an extension of the arrow symbol. Readability clearly IS simplicity.

I would argue that, removing the placeholder makes the unfortunate transition to the "rare case" much more "hacky", which in turn backfires on the original intention of simplifying. In the nominal cases it's not needed anyway so no harm.

[shesek]

Any chance support for decorating the callback function can be added somehow? I regularly use some higher-order functions with callbacks to abstract away some common operations, like handling error delegation with an iferr [1] function. I would really like to be able to do that with backcalls.

Perhaps something like this: ?

iferr cb, (user) <- load_user id
# compiles as
load_user id, iferr cb, (user) ->

[1] https://gist.github.com/shesek/eff0c0abd31ad8457de8

edit: changed to a better example edit 2: this obviously wouldn't work if backcalls works as an expression. do they?

[00dani]

@shesek You can already get that compiled output with the existing backcall design, like this:

contents <- baz qux, foo bar
# actually does not compile as
baz qux, foo bar, (contents) ->
# because it compiles as
baz qux, (foo bar), (contents) ->

Unless you meant something different? (Or unless I've misread the nesting of parentheses-less functions to do something different to what it actually does, which is also possible.)

Edit: Yes, I misinterpreted what baz qux, foo bar, (contents) -> actually means in CoffeeScript. Currently the only way to do that with backcalls is by using some sort of placeholder syntax (or the hug operator, if we go with that). Personally, I think using x <- anywhere that's not the start of the line is a confusing and bad idea, so I don't really think your proposal works so well.

The use of decorators like iferr is a good use case for supporting placeholders, I think:

contents <- baz qux, foo bar, ^
# compiles to
baz qux, foo bar, (contents) ->

It would be better if somehow we could apply decorators without needing an explicit placeholder like that, though.

[shesek]

@00Davo that should compile as baz qux, (foo bar), (contents) ->, which is quite different. For what I meant, the callback should be an argument to foo, an higher order function, which returns a new function that would be used as the callback argument.

edit: (in response to @00Davo's edit) yeah, my proposed syntax didn't feel quite right to me either, and I would prefer to have an placeholder-free syntax for that, too. any ideas for a different syntax, anyone?

[00dani]

It occurs to me that a combination of partial application and composition achieve this:

contents <- compose (baz qux, ...), (foo bar, ...)
# equivalent to
baz qux, (foo bar, (contents) -> *body here*)

That's still pretty verbose and rather unclear, though. Perhaps some sort of concise syntax for partial-application+composition is needed? contents <- baz qux <+> foo bar or something along those lines, maybe.

That idea's getting a bit off-track of backcalls, however, and it's special syntax for a very specific case at present. Thoughts?

[satyr]

What about fat version? <= obviously doesn't work.

[summivox]

If backcalls are not fat by default, +1 for <~ and ~>

[vendethiel]

'Tis a bit too late to change that now, I think ;).

[00dani]

The plan was for backcalls always to produce fat-arrow callbacks, I believe. This seems a reasonable rule, since there aren't any particularly obvious use-cases that require backcalls not to preserve this.

[vendethiel]

any particularly obvious use-cases that require backcalls not to preserve this.

not sure what you mean. That seems to be a fine use case to me :

<- $('img').click
alert @src

[00dani]

I personally don't think a backcall makes sense in that case, @Nami-Doc , mostly because the <- can't be interpreted as "magic =".

In practice that does seem like it might be used anyway, though, so we may need to consider this-preservation in backcalls a little more deeply.

[vendethiel]

Considering we have async everywhere, I don't see why we'd need fat arrows everywhere. I don't need that when I fs.stat; ie ;).

[michaelficarra]

In the proposal, backcalls were assumed to be all "fat" by default. In those uncommon cases where the callback has a meaningful context, either the value is also given as an argument (as in @Nami-Doc's case) or the callback is more of an "event handler" than a continuation (also as in @Nami-Doc's case). In the former case, use a parameter, and in the latter case, use a callback.

[hden]

Consider one use empty lines to improves readability, then what does the following code compiles to?

a <- fn b
c

do d

this

fn(b, function(a){
  c;
  return d();
});

or

fn(b, function(a){
  return c;
});

d();

[michaelficarra]

@hden: The former. Empty lines do not delimit blocks, indentation changes do.

[hden]

@michaelficarra Thanks for clarifying.

So is it discouraged to use backcalls in the out most block, or I'll never be able to outdent back?