Implicit continuation

[00dani]

A syntax to declare a function that replaces implicit returns with an implicit call to a continuation function. Extremely useful for designing continuation-oriented functions, as are ubiquitous in Node.

I'm not sure exactly what operator would be best for this. Here's a few options:

f = (x) ~>
    someTransformationOf x
f = (x) -->
    someTransformationOf x

These would translate to JavaScript like this:

f = function(x, _callback) {
    _callback(someTransformationOf(x));
}

Returns propagated into subexpressions should also become callbacks when using this feature:

f = (x) -->
    if x > 5
        something
    else
        somethingElse

f = function(x, _callback) {
    if (x > 5) {
        _callback(something);
    } else {
        _callback(somethingElse);
    }
}

Explicit returns could also be replaced with callback invocations, although this wouldn't be as important.

[00dani]

I'm not sure how this request is linked to partial application; partial application usually (always?) is used when calling a function, while this is a function-definition-only feature.

The intent here is to provide some flexibility to the implicit-return system, essentially by allowing the return statement to be replaced with an arbitrary function. Ideally, return would be "redefined" to be the callback function within the whole lexical scope, so something like this works:

f = -->
    fs.readFile "somefile", (err, data) ->
        functionThatTransforms data

f = function(_callback) {
    fs.readFile("somefile", function(err, data) {
        _callback(functionThatTransforms(data));
    }
}

The above code may not actually be possible without deeper source-code analysis, since there really should be an implicit return placed before the fs.readFile() call too. That's the general idea, however.

[vendethiel]

I certainly misread your proposal. What happens in the case of an inner callback like you said ? What's wrong with async.js and stuff ?

[00dani]

Not entirely sure how inner callbacks should be handled, because I only considered the idea that implicit-continuation would propagate into callbacks while making that last comment. Here's one possible design, which I've just come up with now and have not tried to apply to any actual scenarios, so it could have serious issues.

Firstly, implicit continuations only ever apply in places where an implicit return would: the last expression of a function. For most expressions, they work just like implicit returns:

-->
    if x
        y
    else
        z

function(_callback) {
    if (x) {
        _callback(y);
    } else {
        _callback(z);
    }
}

(list) -->
    f x for x in list

function(list, _callback) {
    var x, _i, _len, _results;
    _results = [];
    for (_i = 0, _len = list.length; _i < _len; _i++) {
        x = list[_i];
        _results.push(f(x));
    }
    _callback(_results);
}

If the last expression of the function has a callback function, however, then the implicit-continuation replaces returns in that function, and the first function uses normal returns.

-->
    fs.readFile "file", (err, data) ->
        doStuffWith data

function(_callback) {
    return fs.readFile("file", function(err, data) {
        _callback(doStuffWith(data));
        });
}

(I'm rather liking the --> syntax I've shown in the last few examples; it could be called "continued arrow" syntax, which implies the continuation-based stuff it does.)

async.js is an excellent library for using asynchronous functions that take callbacks. Continued-arrow sugar would be more useful for implementing asynchronous callback-taking functions.

[vendethiel]

What's the difference between callback-taking functions and functions that take callbacks ?

[00dani]

Oh, sorry. Nothing whatsoever; I just phrased the expression slightly differently the second time. They're the same thing. The distinction is that async.js is good for using callback-taking functions, while continued arrows are good for implementing them.

[vendethiel]

Then I don't understand your last sentence

async.js is an excellent library for using asynchronous functions that take callbacks. Continued-arrow sugar would be more useful for implementing asynchronous callback-taking functions.

When would that syntax be more useful than async.js

[00dani]

It's not really better; it just does different stuff. Basically, continued arrows allow one to code in continuation-passing style (which Node uses a lot) without sacrificing implicit returns. They don't replace all asynchronous libraries or even most asynchronous libraries.

They're just sugar, inspired by the way implicit returns don't actually seem that useful when a lot of your calls are non-blocking and so their return values don't matter; replacing implicit returns with implicit continuations in appropriate functions restores that functionality.

[jashkenas]

Very cool idea. Thinking it through a bit, I'm not sure that it's fully-baked. It addresses the creating-a-continuation-style-function side of the problem, but doesn't touch the calling-a-continuation-style-function side. It doesn't support idiomatic Node-style callback functions ... and most damningly, the implicit-return-as-value semantics break any callback that needs to be passed more than one value for arguments -- like almost every Node async callback does, with the error as the first argument.

Closing, but I'd love to see a further extension of this proposal that expands and tries to address those wrinkles.

[00dani]

This idea of implicit continuations is really only ever applicable when creating a function, so it naturally isn't going to impact the call side. I think that async.js, in combination with partial application syntax and possibly backcalls, covers the vast majority of calling aspects.

It's true that this proposal as designed doesn't have any way to handle multiple-argument continuations, and that definitely is a show-stopper for Node, unfortunately. I suppose implicit continuation return could be given an array and automatically splat it into separate arguments, but that seems significantly too magic and hence fragile, and when one's already writing out a whole array literal it's hardly much more work just to put the callback function name before it.

Still, it's an idea. Personally, I think partial application and backcalls (are backcalls being added?) are much more important features at the moment, though, since calling async functions is presumably more common than defining them.

[michaelficarra]

Backcalls +1000000! They're the number 1 most important feature this language needs right now. See my coffee-of-my-dreams project for examples. On mobile right now, will post more details later.

[jashkenas]

@michaelficarra -- If you've got a good vision for them, I'd love to hear more details. I've got a bit of hacking time at my disposal.

[michaelficarra]

Taken pretty much from the coffee-of-my-dreams list, the best way to explain my preferred backcall syntax is through example:

• The continuation is captured and tacked on as the final arg by default

  <- fn a
  ...
  b
  

  compiles to

  fn(a, function(){
    ...;
    return b;
  });
  

• The left side of the arrow specifies the argument list for the continuation. When only one is given, parens can be omitted.

  a <- fn b
  ...
  c
  

  compiles to

  fn(b, function(a){
    ...;
    return c;
  });
  

• And for the rare occasion where the continuation is not the final argument of the function, we need to be able to specify where to put it with a marker like the <&> I use below. I'm open to suggestions for a better marker.

  (a, b) <- fn c, <&>, d
  ...
  e
  

  compiles to

  fn(c, function(a, b){
    ...;
    return e;
  }, d);
  

this must be preserved in the continuation by rewriting references as we currently do in bound functions. I don't think we should worry about arguments rewriting, but that's your call.

Here's a real-world example @paulmillr gave in #1942:

compile = (files, callback) ->
  async.sortBy files, @sort, (error, sorted) =>
    return @logError error if error?
    async.map sorted, @map, (error, mapped) =>
      return @logError error if error?
      async.reduce mapped, null, @reduce, (error, reduced) =>
        return @logError error if error?
        @write reduced, (error) =>
          return @logError error if error?
          do @log
          callback? @getClassName()

and here it is using backcalls:

compile = (files, callback) ->
  (error, sorted) <- async.sortBy files, @sort
  return @logError error if error?
  (error, mapped) <- async.map sorted, @map
  return @logError error if error?
  (error, reduced) <- async.reduce mapped, null, @reduce
  return @logError error if error?
  error <- @write reduced
  return @logError error if error?
  do @log
  callback? @getClassName()

[paulmillr]

@michaelficarra actually I changed my mind about backcalls, they would be okay to have around, but they won’t solve main problems of async js. Promises are much nicer and readable:

async.sortBy(files, sort)
  .then (sorted) =>
    async.map sorted, @map
  .then (mapped) =>
    async.reduce mapped, null, reduce
  .then (reduced) =>
    @write reduced
  .fail (error) =>
    @logError error

• Problem 1: errors need to be handled manually by async js. This is something that disturbs me a lot as a node.js user. Monads allow to handle errors in modular and automatic way.
• Problem 2: async code is not composable as simply as monadic code.

[00dani]

Backcalls are monadic, aren't they? The proposed backcall syntax is almost exactly equivalent to Haskell's do notation; it desugars slightly differently because >>= doesn't exist in JS, of course.

The actual problem here, I think, is that asynchronous function calls are one monad, it's a monad that can't do error handling, and there aren't any monad transformers in JS. Adding backcalls doesn't fix that problem, but it's a problem that should be fixable just with some monad-centric libraries; the backcall sugar itself remains valuable for the same reason do notation is.

[epidemian]

That proposal is quite awesome, @michaelficarra =D

There are two little things i'm not sure about though (apart from the problems that i see you now added! =D)

First, if this should be preserved the same way as with functions, i.e. using a thicker version of the arrow, then the bound form of the continuation symbol, <=, would clash with the less-or-equals operator. In fact, the thin version, <-, also introduces some ambiguity, as a <- b is now parsed as a < -b.

Second, i think the example compile function using the proposed callback syntax, while it looks more straightforward, it is kinda confusing with those returns there. I mean they look like normal returns, so at first glance i though they were returning from the compile function (as they don't seem to be in a new scope). But they actually return from another function: that implicit callback that is created with each <-. I think it's confusing that a new scope is created without needing more indentation, IDK.

I was going to mention promises, but it seems @paulmillr beat me to it. Yes, one of the main benefits of promises IMO is that they propagate errors in a sane way (i.e. a composable way).

---

BTW: is there any reason you're calling these functions "backcalls" instead of "callbacks"? I always knew them by the later name, but maybe they are not the same thing :S

[michaelficarra]

@paulmillr: I completely agree, except that solution requires a runtime library. That's not allowed in CoffeeScript.

@00Davo: Yep, backcalls are pretty much exactly Haskell's do. I'm once again trying to slip awesome Haskell features into CoffeeScript without causing a stir by calling it a Haskell feature.

[00dani]

@epidemian, this is assumed to be preserved for all backcalls, which all use <- syntax. There's no clash with <= under this proposal.

[michaelficarra]

@epidemian: But they do end up returning from the entire (back)call stack. I don't see any confusing control flow.

[00dani]

@epidemian This feature is called "backcalls" exactly because the normal sort are callbacks:

someAsyncFunc (x) ->
  # in here is a callback

Backcalls are like callbacks, except the -> function arrow has been flipped around <- (and the location of the callback's arguments is also reversed).

x <- someAsyncFunc
# here is a backcall

[summivox]

Wonder why hasn't anyone brought up https://github.com/maxtaco/coffee-script yet. Not quite "runtime library" if you tell the compiler to inline the iced object. I find await and defer quite natural...

[vendethiel]

Ew. Just look at the generated code.

[paulmillr]

@smilekzs it’s exactly the runtime library. Like Q which I mentioned, but shittier and I doubt it’s as compatible with promises spec as Q. This is ROTTEN:

var a, c, __iced_deferrals, __iced_k, __iced_k_noop, _i, _len,
  _this = this;

__iced_k = __iced_k_noop = function() {};

__iced_deferrals = new iced.Deferrals(__iced_k, {});
for (_i = 0, _len = b.length; _i < _len; _i++) {
  a = b[_i];
  a(b, __iced_deferrals.defer({
    assign_fn: (function() {
      return function() {
        return c = arguments[0];
      };
    })(),
    lineno: 1
  }));
}
__iced_deferrals._fulfill();

[00dani]

Considering @paulmillr 's reference to monadic promises and the fact that backcalls are basically do notation, I'd like the ability to employ, somehow, monad transformers for handling failure while still using the backcall syntax. Basically this would involve reifying the bind operation, as in Haskell where it's >>=, such that bind may be changed for any given backcall stack; bind would default to passing the continuation as a callback but could be replaced with, for instance, a Maybe version that wraps the callback with failure-checking first.

Unfortunately, I can't think of any decent syntax for specifying a custom bind operation. Thoughts?

[summivox]

@paulmillr : I agree the compiled code looks crap. But what's wrong with the await and defer constructs? I mean, bad compiler implementation doesn't really imply bad language.

Correct me if I'm wrong: I see that (deferred) <- f args (as proposed above) is equivalent to await f args, defer(deferred). The defer() construct, however, removes the need to use <&> as proposed by @michaelficarra . (a, b) <- fn c, <&>, d is simply await fn c, defer(a, b), d. I find this more natural than a placeholder, which sound C++-ish to me.

[00dani]

@smilekzs : There's no real way to implement await and defer that doesn't produce horrible CPS compiled output, unfortunately. It's reasonable to have await and defer keywords as real language constructs, like the await keyword from C#, but it's not viable to use them in a situation where your code is expected to compile to readable JavaScript (or to any other readable language).

Note that it would be possible to implement a fairly clean await and defer combo iff they were just a keyword-based version of backcalls, i.e., only working on a single asynchronous call. However, await in IcedCoffeeScript is a block construct, capable of handling an arbitrary number of async calls, and to achieve this it has to pull some really nasty tricks in the JavaScript output.

By contrast, backcalls are a much more intuitive code transformation, and the compiled JavaScript will most likely look exactly the same as it would if one used a normal callback pyramid.

[summivox]

@00Davo : Thanks for the detailed explanation. Actually I find myself mostly using await exactly as backcalls (instead of the block version).

Pity, though. I like the way defer doubles as placeholder--much more consistent, as I've mentioned earlier. Would it be possible for iced-coffee-script to generate the more readable form of compiled code when block-await is absent?

[00dani]

@smilekzs : I haven't actually looked at Iced's codebase, but I think it should be possible for it to detect situations where there's only one defer involved (which, I think, is what makes the difference between a simple backcall-style continuation and the craziness of its full CPS transformation) and produce cleaner code, yes.

However, keep in mind that most callback-taking functions, at least in Node, take the continuation as their last parameter. Backcalls optimise for the common case by not requiring any continuation placeholder at all:

x <- f arg
# equivalent to
f arg, (x) -> # plus indenting the rest of the block
# equivalent to
await f arg, defer(x)

<&> isn't the best placeholder, but it's only needed rarely, so it mightn't matter too much. (Hey, what about using ... as an alternative placeholder, corresponding with its suggested use for partial application?)

[summivox]

@00Davo : I'd suggest a single caret ^, so that you could write:

x <- f arg, ^
x <- f arg #last argument callback
x <- g ^, arg

...which is in line with coffee-script's arrow-rich style ;)

The most ubiquitous usage for <- g ^, arg I could name is setTimeout and friends. But ever since node this is becoming much rarer so guess I could live without the caret(or whatever placeholder).

BTW: That partial application debate has been there for quite some while, but so far I'm not seeing light.

[00dani]

@smilekzs : ^ does already have a meaning in JavaScript (bitwise XOR), but I don't think there's any possibility of conflict between the two uses, and the caret does look good. The visual imagery of an arrow pointing up from the following code (i.e., the continuation) is also effective. I like it.

For setTimeout there's always the option of stealing from Haskell:

flip = (f) -> (x, y, rest...) -> f y, x, rest...
(flip setTimeout) 250, -> 
    do takeAction

[vendethiel]

Would +1 for ... or _.

[epidemian]

Semi on-topic...

This question was asked in StackOverflow a couple of days ago. Basically it asks how to synchronize this kind of code...

fileArray = ['a.json','b.json','c.json']
dict = {}
fileArray.map (f) ->
  fs.readFile f, (err, data) ->
    json.parse data, (k, v) ->
      dict[k] = v

...so it can use the complete dict value once all async calls have been completed.

I wanted to answer with some different alternatives: the "manual" way, the IcedCoffeeScript way and the Futures way.

The "manual" is easy:

fileArray = ['a.json','b.json','c.json']
dict = {}
count = fileArray.length
fileArray.map (f) ->
  fs.readFile f, (err, data) ->
    json.parse data, (k, v) ->
      dict[k] = v
      if --count is 0
        writeToFile dict # Process everything after all callbacks are done.

But i couldn't find an intuitive way to implement the IcedCoffeeScript way (i'll admit that i haven't used it before... so i'm not very used to the constructs). I guess something like this would work:

fileArray = ['a.json','b.json','c.json']
dict = {}
await
  fileArray.map (f) ->
    done = defer()
    fs.readFile f, (err, data) ->
      json.parse data, (k, v) ->
        dict[k] = v
        done()
writeToFile dict

But it still feels quite manual. I mean, assigning defer() to a variable and then calling that explicitly. I'm sure there is a better way to do this in ICS that i'm not seeing.

Also, i'd like to see how backcalls can be used in this case. As far as i can see, they could be used to make the callback code less nested, but a synchronizing mechanism like the counter variable would have to be used to know when all the callbacks were executed, or am i tripping?

[vendethiel]

promises are the future ! :p

[michaelficarra]

@epidemian: you are correct. For that kind of synchronisation, I would use a library like async.js that allows you to specify the intended behaviour a little more clearly than tracking a counter.

[epidemian]

promises are the future ! :p

Ups :blush:

Errata: futures -> promises (though i'm not sure if these are different concepts)

@michaelficarra Ok. Thanks for the pointer :+1:

[summivox]

@michaelficarra : When composability is in concern, CPS seems to fall apart; much like when a few if and for are nested, simple callback pyramid (written either in cascaded form or <- form) falls apart--these situations do require full CPS transformation--even without the counter. @00Davo .

@epidemian : I think this could be a little more idiomatic ICS solution for your example:

# mock-up async for test in browser
readfile=(file, cb)->
  setTimeout (->
    console.log("readfile(#{file}) callback")
    cb(null, file+'data')
  ), 500

parse=(data, cb)->
  setTimeout (->
    console.log("parse(#{data}) callback")
    cb(data, data.charCodeAt(0))
  ), 100

# actual code
fileArray=['a', 'b', 'c']
dict={}

await
  for file in fileArray
    ((file, autocb)->
      # autocb keyword: "return means callback"
      await readfile file, defer(err, data)
      await parse data, defer(k, v)
      dict[k]=v
    )(file, defer())

console.log JSON.stringify dict

So basically when you need to "compose" await blocks, you use

((..., autocb)->
)(..., defer())

which is a form of IIFE actually.

EDIT: this reminds me of Ruby, really.

[00dani]

@smilekzs : That autocb keyword looks an awful lot like the original "implicit continuation" feature request I made up the top there. I think using a different function operator like --> is preferable to a "magic" argument identifier, though.

As for using an IIFE, I assume that'd work with do, like most IIFEs?

await
  for file in fileArray
     do (file, autocb = defer()) ->
       await readfile file, defer(err, data)
       await parse data, defer(k, v)
       dict[k] = v

[summivox]

@00Davo : Yes exactly that. I didn't know do could be used with extra parameters!

What I see is an impasse. Regardless of sugar flavor choices(<- vs. await-defer vs. whatever), it really boils down to whether we could accept convoluted, hard-to-follow compilation output(i.e. CPS output). For coffee-script it's tempting to keep the almost 1:1 relationship with javascript. This way we're left to use async libraries. The other direction is to go evil and ICS is one of the possible results.

My thoughts: (please point out if this is not the appropriate place for these discussion)

• I've never seen anyone complain about their C++ compiler producing incomprehensible ASM. Mostly because we don't DEBUG in ASM most of the time(except for embedded target--which I occasionally do). Were we able to debug in source code, then it doesn't really matter if the compiler output is spaghetti.
• ICS default output is like g++ -g--with debugging information embedded, so if you really want it to be readable instead of debuggable, then I assume these extra info might be safely removed from output. Also, when I debug ICS output, I mentally ignore lines that are not directly generated from my source lines. Not THAT terrible.
• caolan/async, which I use in native javascript(since before I got to know even coffee-script, let alone ICS), hides some of the complexity behind its runtime. That is, code is comprehensible and logically structured, despite boiler-plate-ful. If coffee-script is all about writing concise source code, then re-introducing extra complexity in source seems no-go for me, however readable the output might be.
• For some reason I might have missed, coffee-script dev doesn't seem to like the idea of a runtime. Dilemma: If you don't want the book-keeping code to appear here, it still has to go somewhere, because async code is ubiquitous. If both the compiled solution and the runtime solution are unacceptable, then we're left with primitive sugar that breaks ever so easily.

@michaelficarra : Sure everything comes down to "whether dev likes it or not" but reality is "there ain't no such thing as free lunch". Comments?

[vendethiel]

For some reason I might have missed, coffee-script dev doesn't seem to like the idea of a runtime.

Because of the "it's just javascript". Bad perfs, bad interop, etc

I've never seen anyone complain about their C++ compiler producing incomprehensible ASM. Mostly because we don't DEBUG in ASM most of the time(except for embedded target--which I occasionally do). Were we able to debug in source code, then it doesn't really matter if the compiler output is spaghetti.

But that's not true here, we get on the js a lot. Source maps will help with that but still won't make it disappear.

[summivox]

@Nami-Doc : coffee-script is not entirely runtime-free, strictly speaking. Class support is one such example. Of course you could call these "helpers" but that's exactly the point--being small, concise and with well-understood function, they're allowed.

But that's not true here, we get on the js a lot. Source maps will help with that but still won't make it disappear.

I think I mentioned how I usually debug ICS output. I imagine a what-if situation where we CAN debug directly in source.

[vendethiel]

Usually they're just shims of what we have now but may not on older browsers.

[00dani]

It occurs to me that backcalls, unlike await/defer, don't really "insist" on being used for asynchronous code; backcalls are useful anywhere continuations might be. For instance, backcall syntax might be used in combination with the amb operator as defined in https://gist.github.com/autotelicum/1311262.

[epidemian]

It occurs to me that backcalls, unlike await/defer, don't really "insist" on being used for asynchronous code; backcalls are useful anywhere continuations might be.

Yeap. I was thinking of that too.

One could implement this (quite naive) min function:

minBy = (arr, fn = (x) -> x) ->
  arr.reduce (min, x) ->
    if (fn x) < (fn min) then x else min

As:

minBy = (arr, fn = (x) -> x) ->
  (min, x) <- arr.reduce
  if (fn x) < (fn min) then x else min

(although i'd hope no sane programmer would use backcalls for this kind of thing =P)

Which leads me to the question: what gets returned from a function that uses backcalls (minBy in this case)? I assume in this case it would return the result of reduce. But what would be the rule for implicit returns? "Returns the last expression before any inside-backcall code"?

[vendethiel]

Return the last expression. Coco just calls Call.back to set a @back property on the block.

[epidemian]

Well, i guess you mean that the line with the <- and all its subsequent lines would be considered a single expression, isn't it? I think it makes sense, but at the same time is not so obvious considering that all other multi-line expressions (if, for, etc) use indentation to denote their bodies.

[00dani]

@epidemian : When you get right down to it, all backcalls actually do is not require indentation on their bodies. Were it not for their lack-of-indenting, they'd be entirely equivalent to normal callback syntax. x <- f trivially desugars to f (x) ->, plus indenting the rest of the block.

This is valuable, however, because in effect it turns continuation-passing style code (most async callback functions) back to familiar synchronous code, without the code actually being blocking. The <- operator may be viewed effectively as an async version of =.

Also, another application of backcall syntax occurs to me: RequireJS. (Well, technically this is just another example of backcalls for async functions, but still.)

# without backcalls
require ['jquery'], ($) ->
  require ['someotherlib'], (lib) ->
    codeThatUses $, lib
# with backcalls
$   <- require ['jquery']
lib <- require ['someotherlib']
codeThatUses $, lib

[vendethiel]

Please tell me that you could do this with requirejs

($, lib) <- require ['jquery', 'someotherlib']
codeThatUses $, lib

[00dani]

@Nami-Doc : Looks like RequireJS does indeed work that way, so yes. When using backcalls you wouldn't need to indent the codeThatUses line, though.

Given that edit, yep, that exact code should work.

[vendethiel]

Oops, I typed too fast. Fixed.

[epidemian]

@00Davo yes, i get what the benefits of backcall syntax are. I was mostly trying to understand how would they work in functions whose return values are important. On asynchronous functions the return value is usually not important, the arguments passed to their callbacks are. But on synchronous function that just happen to use callbacks (e.g. Array::map and friends, Array::sort, etc), the return value of the function is usually important, so if backcalls were to be used in these cases too, then i'd expect the rules for their return values would not be very confusing.

That being said, the example of RequireJS is a great use-case for backcalls :smiley:

[00dani]

@epidemian : Backcalls are a very simple syntactic transformation to the equivalent callback structure; they don't have any impact on implicit returns, so your example using reduce will work in precisely the same way as the ->-based version. It would be a very strange way to code the function, though.

There is one other difference between -> and <- that may be worth noting, although it doesn't affect your example: Backcalls are treated as => functions instead of -> ones, i.e., they preserve the value of this from outside the backcall function. As far as I can tell this is what you'd want for the vast majority of backcall uses, which is good because <= is already taken.

Another thought: Does anyone see value in a backcall version of do, either just making <- do (and related stuff like (x = 5) <- do) work or figuring out a different keyword for it?