Feature Request: Every async function returns Promise

[rdner]

Now in node we have callback or EventEmitter model to deal with async calls by default. But in my opinion it is better if every async function returns a native Promise (from new version of V8). It does not break backward compatibility and supports optional callback if needed.

If so we just do not need to install additional package for promises like q or bluebird except additional functionality is needed.

---

_EDIT 2014-12-11 by @rvagg_

Comment lifted from here so it's easier to see for newcomers to this conversation.

This was discussed at the TC meeting yesterday, see #144, the aim was to be able to provide at least some kind of statement as feedback in this issue. I don't think the issue needs to be closed and can continue to collect discussion from those who feel strongly about this topic.

The feedback from the TC about incorporating a Promises-based API in core goes something like this:

A Promises API doesn’t make sense for core right now because it's too early in the evolution of V8-based promises and their relationship to other ES* features. There is very little interest within the TC in exploring this in core in the short-term.

However, the TC is open to change as the feature specifications and implementations in ES6 and ES7 are worked out. The TC is open to experimentation and providing the most optimal API for users, which may potentially include a Promises-based API, particularly if newer features of JavaScript work most optimally in conjunction with Promises. The speed of the language specification process and V8 implementation will mostly dictate the timeline.

It should be noted that a callback API is unlikely to ever go away.

---

[phpnode]

I'm saying don't do that, the right way to do this is at the call site,

let promise = Promise.resolve(someSyncFunction(123)):
promise.then(console.log.bind(console));

[benjamingr]

@bjouhier you don't and you generally shouldn't.

Passing the promise constructor places is silly and should generally be avoided. I merely stated that cast is deprecated not sure why you'd assume me or @phpnode were in favor of passing anything related to promises to sync functions.

If a function is synchronous converting it to a promise returning one is at best pointless.

[bjouhier]

Turning sync functions (like fs.readFileSync) into functions that return a promise is not interesting.

What's interesting is to turn callback-based async functions (likefs.readFile) into functions that return a promise. This can be done with a simple API principle: if the function is called without any callback it returns a promise.

Note that the promise itself is returned synchronously, but the returned promise is resolved asynchronously.

[phpnode]

@bjouhier I think you've misread what @benjamingr and I are saying, we agree with you. We're responding to @zowers suggestion

[bjouhier]

Makes complete sense now. Thanks.

2015-02-03 23:38 GMT+01:00 Charles Pick notifications@github.com:

@bjouhier https://github.com/bjouhier I think you've misread what @benjamingr https://github.com/benjamingr and I are saying, we agree with you. We're responding to @zowers https://github.com/zowers suggestion https://github.com/iojs/io.js/issues/11#issuecomment-72623713

— Reply to this email directly or view it on GitHub https://github.com/iojs/io.js/issues/11#issuecomment-72751203.

[bjouhier]

The crypto functions are a bit of an exception in node API, and they bring confusion in this debate. I think that the design should focus on standard node APIs (like fs). Then one can worry about fixing crypto.

2015-02-03 23:40 GMT+01:00 Bruno Jouhier bjouhier@gmail.com:

Makes complete sense now. Thanks.

2015-02-03 23:38 GMT+01:00 Charles Pick notifications@github.com:

@bjouhier https://github.com/bjouhier I think you've misread what @benjamingr https://github.com/benjamingr and I are saying, we agree with you. We're responding to @zowers https://github.com/zowers suggestion https://github.com/iojs/io.js/issues/11#issuecomment-72623713

— Reply to this email directly or view it on GitHub https://github.com/iojs/io.js/issues/11#issuecomment-72751203.

[domenic]

One idea that has been brought up, which I don't necessarily agree with but wanted to make sure people see, was to gate changes on the use of ES6 module syntax and loader. It's unknown when that would land. But maybe if you do import { readFile } from "fs"; you get a promise-returning function, whereas if you do const { readFile } = require("fs");, you get a callback-accepting function.

This is probably not necessary for standard modules like fs. But maybe it could be interesting for crypto.

---

What I am most interested in is whether we can gain any performance from more directly using V8 PromiseResolver objects directly, instead of using Node's MakeCallback infrastructure and then wrapping that into a Promise in user-land. (Neither will likely be as fast as callbacks, simply because V8 promises are slow as all hell. But that's a pretty separate problem.)

[benjamingr]

What I am most interested in is whether we can gain any performance from more directly using V8 PromiseResolver objects directly.

From what I understand if you can pass the context to a resolver you can make interesting gains, not sure if you've seen https://github.com/iojs/io.js/issues/704 . As said several times earlier in this thread - avoiding allocating a closure is key if we want native promises in code.

[bjouhier]

@domenic Priority 1 should be to ensure that callbacks will continue to work as before, and that their performance will not be impacted. This is easy to achieve. Priority 2 is to close the performance gap between promises and callbacks (while preserving priority 1 of course). More challenging. We are probably all aligned on this but I just wanted to stress it because it is key to get wide acceptance of a dual API.

The import vs. require solution feels a bit hacky (or too clever).

[domenic]

@bjouhier agreed on all counts, including "a bit hacky (or too clever)" :)

[rvagg]

The no-callback = return a Promise feels a tiny bit magic but perhaps will feel natural as native Promises start to infect everyone's code. My main concern about this pattern is the lack of visibility, or explicitness about what's going on. You're not saying you want a Promise, you could just have forgotten to add a callback or perhaps you wanted a Sync method but made a typo and then it wouldn't be easily caught like it currently is at runtime.

This concern would be partially alleviated by the unhandled rejection hook proposed in #256 (or similar).

[domenic]

@rvagg we definitely should get the unhandled rejection hook in. But yeah, I kind of agree that dual APIs are icky---even as a promise user I'd prefer no silly callbacks infecting my code. Any arguments I pass after the usual ones should be ignored! :P.

It's probably the best solution though, because otherwise you have to make promises second-class somehow (const { readFile } = require("fs/as-promised") or const { readFile } = require("fs2") or const { readFile } = require("p/fs") or const { readFileP: readFile } = require("fs") or...).

[rvagg]

what's the argument against just going fs.readFileP() or fs.readFilePromise()? is it just that it feels second-class?

[qfox]

Explicit is better than implicit. +1 @rvagg

[Fishrock123]

Explicit is better than implicit.

fs.readdirCallback(...)

[qfox]

@Fishrock123 I'm fine with that ;-) Also I know that you can't get speed and runtime arity detection at once. It should be done at v8 side.

Also I think that there are async/await + generators and API will change anyway. We can get a pause and make readdirPromise and/or readdirCallback ;-)

[benjamingr]

@zxqfox explicit is better than implicit readdirTakesPathStringReturnsPromise :D This is fun @Fishrock123

[Fishrock123]

Also I think that there are async/await + generators and API will change anyway.

async/await will use promises. Having this be second-class is going to be a problem in the future. it's not even hypothetical anymore.

[qfox]

@Fishrock123 Absolutely. But will. Atm we have prototypes and polyfills like https://github.com/jhusain/asyncgenerator and we need v8 support for it. Don't it?

[Fishrock123]

Yes, no javascript engine currently supports it. The spec is in drafting. But it is becoming spec, and we do need to be attentive to that too. :)

[benjamingr]

@zxqfox @Fishrock123 woah easy there - asyncgenerator is still at an early stage yet (compared to async/await alone, or generators alone) and doesn't even have real support in transpilers yet as far as I know (the for.. on loop) - not to mention it talks about observables - which replace streams and not callbacks. I think it's at least "a year or two" before we should be even talking about the possibility of using it in node core :)

[phpnode]

It's possible to use async/await with e.g. 6to5 today, and it really makes things a lot nicer. It is completely obvious that this will become the preferred style, so it makes sense to have first class promises, even if certain members of the community don't like it. Needing to append Async or P or Promise to everything really sucks.

async function loadJSON (dirname) {
   let filenames = await fs.readdirAsync(dirname);
   let items = {};
   for (let i = 0; i < filenames.length; i++) {
     let filename = filenames[i];
     items[filename] = JSON.parse(await fs.readFileAsync(path.join(dirname, filename), 'utf8'));
   }
   return items;
}

vs

async function loadJSON (dirname) {
   let filenames = await fs.readdir(dirname);
   let items = {};
   for (let i = 0; i < filenames.length; i++) {
     let filename = filenames[i];
     items[filename] = JSON.parse(await fs.readFile(path.join(dirname, filename), 'utf8'));
   }
   return items;
}

[qfox]

@benjamingr I've put it here because it's a suggestion of streams, all hates current stream API, and it's a thing than can fix current realization. In that case Stream API will change, users will have to learn that new API (again, yes), and probably all other API could be fixed as well.

Anyway. Let's suggest that promised API will be 0.1% slower than current API (It's very optimistic but still it can be very optimized, why not?). Should we replace the current implementation with them in that case? Anyway it's slower and it can't be same speed, right? Or we don't care about speed?

If we want to have a first-class promised libraries we should left callbacks somewhere for guys who don't want to rewrite their code or want faster realization. As well as we using while or for for arrays and array-based queues instead of forEach or reduce. Anyway, promises, generators, etc are sugar. Sure it looks nicer.

Let's rename old libraries to *-cb and make new shiny as replacement. Also we can have a transition major version with fs-p and fs-cb both allowed at once.

[domenic]

@zxqfox although that suggestion is aesthetically appealing and I would enjoy it personally, it just seems to have no practical benefits over a coexistence strategy, and instead will just cause lots of pain for people.

[qfox]

@domenic Anyway we need both. Sugar for sweeties and callbacks for speed. We can't get both in one package.

So we can get a break to make the fastest promises realization, compare speed with callbacks, and decide something.

I'm not really against head&shoulders, but I don't believe that it will be as fast as current callbacks. ;-( Unfortunately.

[benjamingr]

@zxqfox there are userland promise libraries that outperform callbacks right now since allocating a closure is more expensive than allocating a promise object (just ask @petkaantonov ). I don't see any reason native promises wouldn't cross that barrier eventually.

A lot of people don't like a lot of things but breaking APIs without an extremely good reason is a big no-no IMO. Let's not jump above our heads here with streams.

[pesho]

Regarding speed, isn't the current implementation already forced to check whether typeof callback === 'function'? Or does it just go ahead and attempt to call an undefined...

[phpnode]

@pesho yeah, speed is a total red herring here, in fact the current implementation does a load of work which would not be necessary if it could just return a promise instead: https://github.com/iojs/io.js/blob/v1.x/lib/fs.js#L59

[qfox]

@benjamingr I'm just counting processor ticks ;-) More operations = more ticks. Isn't it? I hope he will come and clarify the reason.

http://spion.github.io/posts/why-i-am-switching-to-promises.html I don't know how to get sources of that benchmark but still there are things much faster than bluebird promises.

@phpnode It's a good sign to make a promised API and to compare speed.

[benjamingr]

@zxqfox I really don't want to make this about bluebird - but it's a lot faster now than it was in those benchmarks more than a year ago and I'll reiterate again: You need less ticks to allocate a promise than to allocate a closure when passing a new function as a callback.

No one is suggesting a userland promise library in core - the point is given time and cooperation there is no reason (native) promises in core would not be at least as fast for the average user as callbacks. Speed shouldn't be an issue here.

[drewhamlett]

Honestly, I wouldn't care if Promises were 50% slower. I'd still use them.

While you guys are making your brain explode, the rest of us we'll be using async / await. Good luck.

[qfox]

@benjamingr So you won't allocate callbacks, right? Even in .then, .catch? Can't get your point. To clarify: I'm not against "return promise if there are no callback". It's not a holy war between closures and promises for me. Just imho better to have both. And I can fix names in my old code if needed.

Just:

• Do we need to modify current function for back-compatibility only?
• Can we move current functionality to *Async and replace originals with promised version?

We have 2 PRs (or more) already and the second one is a nice try. Not finished yet though.

@drewhamlett Same here. But sometimes don't. It's like using a truck to carry a spoon to your neighbour.

[petkaantonov]

@zxqfox no you wont allocate any closure when consuming a promise with await or yield. Whether that allocates a closure in the background depends on your runner (bluebird.coroutine doesn't so its not impossible).

[qfox]

@petkaantonov Thanks for clarification. It makes sense.

[seidtgeist]

@petkaantonov By the way, are you aware of any sensible macro or cross-compilation goodness for async and await?

[pesho]

By the way, are you aware of any sensible macro or cross-compilation goodness for async and await?

6to5 can convert async/await to Bluebird.coroutine(): http://6to5.org/docs/usage/transformers/#bluebird-coroutines

[mikeal]

I don't think this whole "when the callback is omitted a promise is returned" thing will work because several APIs have optional parameters in the middle. Example: https://iojs.org/api/fs.html#fs_fs_symlink_destination_path_type_callback

[mikeal]

@domenic if Promises are super slow in v8 right now I assume there are plans to fix that. I'd be hesitant to rely on some of the internals you're suggesting before that performance work is done given v8's tendency to change API.

[domenic]

@mikeal that is not an issue.

fs.symlink(dest, path, type) // returns promise
fs.symlink(dest, path, type, callback) // uses cb
fs.symlink(dest, path) // returns promise
fs.symlink(dest, path, callback) // uses cb

[mikeal]

@domenic how exactly are you detecting that that callback is omitted in those cases? A length check and a type check?

[domenic]

@mikeal exactly like we do right now https://github.com/iojs/io.js/blob/3e675e44b59f1be8e5581de000f3cb17ef747c14/lib/fs.js#L809-L810

[mikeal]

@domenic this feels a little too magical, why is this preferable to require('fs').promisfied()?

[domenic]

@mikeal the same reason "use strict" is a bad thing to ask of people. You're basically saying "please add this magic incantation everywhere if you want to be modern." In the end it just looks like a vestigial ugly thing that reminds us "lol iojs has so much legacy, I have to put .promisified() every time I require a core module."

[mikeal]

@domenic I guess I find that being explicit is preferable to changing the behavior this dramatically based on argument length. Think about the failure case here, if you're using a callback and forget an argument you're getting a promise returned that you don't do anything with. If you want the promise API you're going to get a property access error on the return value if you accidentally send an extra argument.

I've been using Node for a very long time and I routinely forget arguments :smiley:

I find your proposal for exposing a new API by default when using the new module spec preferable. In fact, I think that we should lump all the changes in behavior we would want to make in to that transition.

[Fishrock123]

Think about the failure case here, if you're using a callback and forget an argument you're getting a promise returned that you don't do anything with. If you want the promise API you're going to get a property access error on the return value if you accidentally send an extra argument.

Just check if the last thing is a function in most cases?

[qfox]

@domenic Can we vote somehow? ;-)

I find your proposal for exposing a new API by default when using the new module spec preferable. In fact, I think that we should lump all the changes in behavior we would want to make in to that transition.

:+1: yes, please

[greim]

...if you're using a callback and forget an argument you're getting a promise returned that you don't do anything with...

Tell me if I'm reading that code wrong, but I don't think forgetting any number of optional middle arguments would cause a promise to be returned with that code. And if someone forgets the callback, they've got bigger problems than allocating unused promises.

If you want the promise API you're going to get a property access error on the return value if you accidentally send an extra argument.

Only if the extra arg is a function, right? I'd think a bigger worry would be newcomers having naive-but-perhaps-reasonable expectations that both would work simultaneously. Which might be an acceptable level of confusion anyway since no matter how this shakes out it will leave a legacy quirk that newcomers must grapple with.

[19h]

:-1: I'm against adding Promises to the core functions. They are a second class abstraction layer and should be kept what they are: user land sugar. Promises blur the scope of our core API and increases the maintenance effort. That given, I want you to focus on overhead and performance. Native Promises are far from perfect; they are slower than a user-land implementation (bluebird, that is - by a factor of 4) and await is not even standardized at all.

Lets keep the core clean and fast. Everything else is up to the user.

[sam-github]

The intention isn't to merge promises next week. Its to experiment with them, see the pros and cons (such as @mikeal 's concerns about confusion), see how V8 support having native promises changes the landscape (it it does), and see how the coming flow control (such as await) changes the async landscape when it comes.

If js devs start routinely using Promises and await iojs will look creaky and ancient if it doesn't support them. At which time, it will be good to have a plan. If that doesn't happen, having some solid benchmark data and experience with what the implications are of promises in core will be good.

Its also possible that some small changes in core will make it easier to implement promisification out-of-core. This is also an ongoing area of experimentation.

[benjamingr]

+1 for everything Sam said. Io needs a clear roadmap to promise support and experimentation is positive here. Native promises still have several issues to solve but we want them in when they do JavaScript 2015 is just 4 months away