Thoughts on Q.async

[ForbesLindesay]

This is just a thought, feel free to disagree with me, but I'd just like to throw the idea out there.

I have a suspicion, that once yield is available in more environments (I'm currently writing a shim to compile it to ECMA5 based on the current standards) we'll see lots of methods that look like the following.

var sumOfStuff = Q.async(function(a,b,c,d,e,f,g,h){
    a = yield a;
    b = yield b;
    c = yield c;
    d = yield d;
    e = yield e;
    f = yield f;
    g = yield g;
    h = yield h;
    var other = yield someRemoteData({id:a});
    Q.return(other+b+c+d+e+f+g+h);
});

That is to say, lots of functions will begin by yielding on all their arguments, either one at a time, or using something like Q.all.

What might be preferable, would be to resolve all promises that are passed as arguments, before giving them to the function. The only down side, is that it prevents you from writing functions like:

var hash = Q.async(function(pass){
    var salt = getRandom();//computationally expensive
    Q.return(salt + yield pass);
});

Perhaps we could have some way of annotating the function as lazy but resolve all arguments by default?

[domenic]

I don't think this makes much sense, simply because we have no idea which of the arguments should be resolved in parallel versus not. I.e. should that be

let sumOfStuff = Q.async(function* (a, b, c) {
    let [A, B, C] = [yield a, yield b, yield c];
    // ...
});

or

let sumOfStuff = Q.async(function* (a, b, c) {
    let [A, B, C] = yield Q.all([a, b, c]);
    // ...
});

or

let sumOfStuff = Q.async(function* (a, b, c) {
    let [A, B] = yield Q.all([a, b]);
    let C = yield c;
    // ...
});

or even

let sumOfStuff = Q.async(function* (a, b, c) {
    let [A, B, C] = yield Q.allResolved([a, b, c]);
    // error recovery code for rejected A, B, or C
    // ...
});

[ForbesLindesay]

For a well behaved promise, does it actually make a difference whether they're resolved in serial or parallel? I would envisage error handling code for arguments having to sit outside the function:

let sumOfStuff = Q.async(function* (a, b, c) {
    // ...
}).fail(function(err){
    //error recovery for rejected parameters
});

but error recovery is a good argument for doing it the manual way, I hadn't thought of that.

[domenic]

For a well behaved promise, does it actually make a difference whether they're resolved in serial or parallel?

You are right, sorry. I was confusing this with the case of calling multiple promise-returning functions.

It still seems likely that there are significant differences, since each yield is a suspension of the execution state of the function and then a reentry later. Perhaps it would simply be a matter of n nextTicks for n yields. I'd have to think harder about how generators and Q.async work to be sure.

In that case it sounds like the convenience method you're asking for could be implemented as

Q.deepAsync = function (generator) {
    return Q.async(function *() {
        var args = Array.prototype.slice.call(arguments);
        var argsResolved = yield Q.all(args);
        return generator.apply(this, argsResolved);
    });
};

(off the top of my head, untested, might be buggy, etc. disclaimer)

[kriskowal]

@gozala has previously recommended (privately) that we add Q.promised as a function decorator that guarantees that the return value is a promise, and guarantees that all arguments are fulfilled before calling. This is a function that he uses in his work at Mozilla. I am hesitant because, for remote objects, using "when" actually ought to be very uncommon, favoring the message passing forms, like "get" and "post". It would be common to pass an unresolved promise to a function and for the function to interact with the unresolved promise through asynchronous message passing.

That probably does not diminish the utility of a wrapper for synchronizing arguments. I will entertain the introduction of Q.promised. I’m not sure about the color though. Does anyone have a better idea for the name?

[kriskowal]

Let’s close this issue and open a new issue for an orthogonal Q.promised function decorator.

[Gozala]

To be more precise we use minimalistic Q subset with addition of promised decorator in Add-on SDK: https://addons.mozilla.org/en-US/developers/docs/sdk/latest/packages/api-utils/promise.html

Idea behind is just to ease expression of computation on promise values. So instead of having utilities like Q.all we just expect users to use promised(Array).

I have no idea how that would work with remote promises, but I guess in exact same way as Q.all would.

[domenic]

@kriskowal re: remote objects and get/post/etc., I've always thought something like this gist was the way to go. Not sure if anyone ever actually turned that into a library, although it looks like @Gozala's meta-promise is a (SpiderMonkey-specific?) realization of it.

This doesn't really address your concern, but perhaps indicates that Q.async + Q.promised + meta-promise--like proxy promises could be mashed up into an awesome remote-promises--friendly wrapper.

[Gozala]

@domenic meta-promise was experiment that should be pretty easy to update to make it compatible with all JS engines supporting Proxies. Although from that experience I learned that making promises too much like regular objects was very confusing. I think syntax in the ES proposals http://wiki.ecmascript.org/doku.php?id=strawman:concurrency is probably a best way to go about it:

files!filter(function (name) {
  return (name.slice(-3) === '.js');
});

Makes it obvious for reader that operation happens eventually rather then now.

I also have experimented with that approach in clojurescript where you have much more control of a language https://github.com/Gozala/eventual-cljs and seem to like it the most so far.

[domenic]

@Gozala Without new syntax, what about something like files.eventually.filter(...). I'm doing something similar in Chai as Promised.

[Gozala]

@Gozala Without new syntax, what about something like files.eventually.filter(...). I'm doing something similar in Chai as > Promised.

I guess that may work, but don't know if it's distinct enough for users to spot though. Still I think promised decorator is kind of more obvious and works on js engines today, althouh it's not very compatible with OOP style. But since I tend to go functional most of the time following worked extremely well:

promised(filter)(function(name) {
  return (name.slice(-3) === '.js');
}, files)

[Gozala]

@domenic BTW if you have access to generators than you can use yield to wait for a promise resolution. I have played with that idea quite a while ago: https://github.com/Gozala/actor

[Gozala]

Just recalled that @dherman has a well maintained library http://taskjs.org/ that implements very similar idea

[domenic]

@Gozala Haha yes yield is how how this whole thread got started :). But I was thinking of the remote promises case where you don't want to actually wait for resolution until the last minute.

[Gozala]

@domenic Oh BTW as of https://gist.github.com/1372013 I think most of the things you do there are better of in the streams land, in my opinion plain promises are not well suited for representing sequential values. Although you can build streams using promises which I tried https://github.com/Gozala/streamer/wiki/stream

[kriskowal]

The way remote promises should work (they are presently broken in Q-Comm) is that the promise will be locally resolved in ½ RTT from the remote resolution. The local resolution will have the promise API for passing messages to the remote promise. Like this:

var remote = getRemotePromise();
remote.get('a') // works here, and will resolve in ½ RTT of remote resolution
remote.then(function (remote) {
    remote.get('a') // will resolve in 1 + ½ full RTT after remote resolution
   // a minimum of 2 round trips from the previous event
})

As you can see, "when" will always introduce latency by waiting for synchronization. The only reason to wait for a remote promise is to way for synchronization side-effects, which should be relatively rare for performance reasons.

[Gozala]

In fact we don't implement Q.when and do resolve promises in the same turn of event loop, mainly because some capabilities are only exposed across the call stack of the event handler and attempt to use them in next turn will fail.

[ForbesLindesay]

I like the idea of having a separate Q.promised function, it should be simple enough to wrap

var function = Q.promised(Q.async(function* (arg1, arg2){

}));

So I think that's a good solution. Remote objects are a very different scenario. My initial thought would be to stick to the current get, invoke etc. but let people easily extend that by adding their own functions which just call into those base functions.

If you're looking at processing lists, I think it's well worth considering libraries like Reactive Extensions. Lists are very different to promises, and if we want to do work on remote lists and apply things like remote filter functions, we should consider a separate library (and learn from things like LINQ to SQL). I do think many of these libraries may benefit from an 'all' method, which returns a promise for the complete list/stream returned as an array.

Just my 2 cents worth.

[domenic]

Closing in favor of #87