Syntax proposal: async/await

jaredly commented 7 years ago

Except using lwt instead of promises? orr we could not tie it to anything in particular and have it construct calls to async_map fn (val) so you could do let async_map = Lwt.map at the top of your file, or let async_map fn val = Js.Promise.then_ (fun v => Js.Promise.resolve (fn v)) val. Or maybe it should transform to bind instead of map?

jordwalke commented 7 years ago

Sounds good! Perhaps defining something like

module Async = Lwt

At the top of the file could be how you select the async "provider". I really like Lwt's syntactic ppx plugin, and we could provide first class syntax for it.

jaredly commented 7 years ago

Hmmm actually I'm thinking that all I really need is ppx_let https://github.com/janestreet/ppx_let with the proper setup. Unfortunately it looks like reason doesn't support the 4.03 let%ext syntax :(

cullophid commented 7 years ago

does this not require a decision on what the standard for async should be ? I was under the impression that that was still not decided...

jaredly commented 7 years ago

Actually, I've got a ppx together that isn't tied to any specific async implementation, as long as it adheres to a general signature.

brentvatne commented 7 years ago

@jaredly - have you thought more about this since July? I'm pretty interested in async/await syntax in reason -- the current async tools are a detractor for people coming from the JS side if you're used to having async/await. It would certainly make it an easier sell to re-write some of Expo's existing JS with Reason.

jordwalke commented 7 years ago

@brentvatne This has been at the top of our minds this week. There's a couple of approaches. Currently, I'm trying to weight the pros/cons between supporting syntactic extension for promise libraries like Lwt, vs. providing syntactic sugar for callbacks.

Syntactic sugar for callbacks would not full featured, and has some pitfalls - however, it is much lighter weight by default, and is simply sugar for supplying a final callback argument: "Lightest Weight Threads" has a nice ring to it: Any function that accepts a callback of some regular form would be compatible.

You could imagine the let! keyword being used to indicate an async computation.

let asyncIncrement = (i, andThen) => andThen(i + 1);

let myFunc = (initialVal) => {
  let! foo = asyncIncrement(10);
  let z = nonAsync(20, foo);
  lat q = asyncIncrement(z);
  q;
};
let! finalValue = myFunc(200);
print_int(finalValue);

More examples, including how let! / and could enable "parellel" tasks (concurrent, actually):

https://gist.github.com/jordwalke/1cd1f18ef060e2a5226f54f31417f6f2

We could instead build syntactic support for integrating with fully featured promise libraries such as Lwt by using ppx_let. I wonder if the two approaches are not incompatible. I wish there was a way for one to progressively enhance to the other.

kay-is commented 7 years ago

I liked the Backcalls of LiveScript.

brentvatne commented 7 years ago

@jordwalke - the "Lightest Weight Threads" api looks elegant and easy to use. Coming from the JS side, my main concern is that when integrating Reason code with existing JS libs we'd want to be able to use this syntax with functions that return Promises, ideally without having to manually wrap them with another function for interop.

Suppose asyncIncrement returns Js.Promise, then let! foo = asyncIncrement(5) would have a similar desugaring but rather than passing in a callback to asyncIncrement it would pass it in to Js.Promise.then_. Haven't thought about how to handle .catch though.

jordwalke commented 7 years ago

@brentvatne Yeah, we'd want to create a way to interop with full featured promise libraries (including JS promises).

You could specify a promise provider perhaps:

let myFunction = () => {
   open Promise;
   let! thisIsTheResultOfJSPromise = asyncCall();
   thisIsTheResultOfJSPromise + 10;
};

Or for short:

let myFunction = () => {
   Promise.let thisIsTheResultOfJSPromise = asyncCall();
   thisIsTheResultOfJSPromise + 10;
};

By default, you might consider that there's this hypothetical open Callback by default which uses callbacks for the default behavior. (Or maybe the build system per project could decide the default).

Module.let has the visual affordance of "Library Defined Let".

hcarty commented 7 years ago

I really like the idea of Module.let as it makes things very explicit and, when useful, makes it cleaner and more readable when multiple modules are to mixed - M1.let ...; M2.let ...; ....

A ! suffix on keywords is already used to signify that something is being overridden. It would be nice to make that use consistent, either by choosing something else as the let suffix or changing open! and friends to use a different syntax. That said, it may be nice to save a sugared let! until the language can support it more cleanly with modular implicits or something similar.

hcarty commented 7 years ago

For any syntax it would be nice to support the same for other keywords like switch, try, and if. Lwt's ppx supports this and I think ppx_let does as well.

Lwt.switch promise {
| A => ...
| B => ...
| exception Not_found => ...
}

cullophid commented 7 years ago

why not just use the same api as in js ?

let async f = () => {
let f = await somePromise();
someFunct(f);
}

This would require a standardized promise implementation.

andreypopp commented 6 years ago

I propose adding two syntax constructs which could be described as syntax sugar for ppx_let and therefore can used with any monadic structures (promises, option, result, ...).

let-do-bindings: let N = do E;
do-expressions: do E;

Syntax

Let-do-bindings

Reason:

let N = do E;
M

OCaml:

let%bind N = E in M

OCaml (after preprocessing with ppx_let):

Let_syntax.bind ~f:(fun N -> M) E

Let-do-bindings are used when you need to exract a value form a monad and define some new computations with it. For promises: "wait for a promise to be resolved, get its value and return a new promise based on it".

Do-expressions

Reason:

do E;
M

OCaml:

let%bind () = E in M

OCaml (after preprocessing with ppx_let):

Let_syntax.bind ~f:(fun () -> M) E

Do-expressions are used when you need to perform a series of side-effectful computations for which result values are not defined (they are represented as unit () and so you care only about the side-effects). For promises: "perform a series of async actions on a filesystem: copy, move files".

Using syntax with concrete monads

ppx_let desugars let%bind into Let_syntax.bind calls. That means Let_syntax module should be in scope when using the proposed syntax constructs.

I like how we can use local module open to bring the needed Let_syntax into scope:

module Promise = {
  module Let_syntax = {
    let bind = (~f, promise) =>
      Js.Promise.then_(f, promise);
  };
};

let getJSON = (url) => Promise.({
  do AsyncLogService.debug("getting data...");
  let resp = do fetch(url);
  let data = do resp.json();
  Yojson.Safe.parse(data);
});

Parallel bindings

let_ppx supports "parallel"-bindings:

let%bind x = getJSON("http://example.com/x.json")
     and y = getJSON("http://example.com/y.json")
in ...

This maps on let-do-bindings:

let x = do getJSON("http://example.com/x.json")
and y = do getJSON("http://example.com/y.json");
...

The downside (?) is that do keyword is required for each of the bindings in a group. At the same time I don't think that allowing to omit it for and N bindings is a good idea — it will bring less clarity.

Note that ppx_let implements parallel bindings in terms of Let_syntax.both function. You can read more in its README.

Adding an implicit return value for the last do-expression

Currently in a chain of do-expression, the last value must be some monadic value. For example, with promises:

let copydir = (from, to) => Promise.({
  do Fs.copy(from, to);
  Js.Promise.resolve(());
})

We can add such return value implicitly in terms of the current Let_syntax in scope.

ppx_let doesn't have this feature so I'd describe it mapping from Reason to preprocessed OCaml code directly.

Note the refined definition of Promise.Let_syntax which includes return function to wrap any value into a promise.

Reason:

module Promise = {
  module Let_syntax = {
    let bind = ... as defined before ...

    let return = (v) =>
      Js.Promise.resolve(v);
  };
};

let copydir = (from, to) => Promise.({
  do Fs.copy(from, to);
})

OCaml:

let copydir from to = Promise.(
  Let_syntax.bind
    ~f:(fun () -> Let_syntax.return ())
    (Fs.copy fromb to)
)

Async-specific syntax vs. Monadic syntax

Simply, I believe async-specific keywords don't bring much value given that do-keyword already fits well with async computations (warning: I'm not native speaker, I might be totally wrong about that).

The advantage is that we can use the proposed syntax with other monads than async computations:

option - computations which may result in values which are absent
result - computations which may result in errors
... other custom monads

The async computations use case already requires us to provide different "bindings" to the syntax. We want to define async computations at least with 3 different implementations:

Js.Promise
lwt lib
async lib

ppx_let approach solves that elegantly with Let_syntax-convention.

jordwalke commented 6 years ago

Good discussion. Here's some thoughts/questions:

When would the value automatically be wrapped in return and when would it not? How would you opt out of that behavior?
The word do does imply side effect imho. It might not be the right word for things like async/option monads.
It's nice how do is one keyword whose position alternates its meaning between bind and imperative bind.

andreypopp commented 6 years ago

When would the value automatically be wrapped in return and when would it not? How would you opt out of that behavior?

The only case, I can think of, which is useful to introduce an implicit return is when do E; is the last expression:

{
  do E;
}

In that case instead of parsing it as:

let%bind () = E in ()

we parse it as:

let%bind () = E in [%return ()]

which is then transformed into:

Let_syntax.bind ~f(fun () -> Let_syntax.return ()) E

To opt-out — simply add a desired expression after the do E;:

{
  do E;
  return(result);
}

Now the question is how to refer to the current return, the obvious answer is via Let_syntax.return which is in scope already (for ppx_let). That might be noisy so we potentially could extend the Let_syntax convention to have a module Let_syntax.Open_in_body which we will be opened in body of the monadic expression:

let N = do E;
M

will be transformed into:

Let_syntax.bind
  ~f(fun N -> let open Let_syntax.Open_in_body in M)
  E

Similar thing for do-expressions.

The complete example would look like:

module Promise = {
  module Let_syntax = {
    let bind = (~f, promise) =>
      Js.Promise.then_(f, promise);

    module Open_in_body = {
      let return = (v) =>
        Js.Promise.resolve(v)
    }
  };
};

let getJSON = (url) => Promise.({
  let resp = do fetch(url);
  let data = do resp.json();
  return(Yojson.Safe.parse(data));
});

The downside is that return will appear magically after the let-do or do syntax. It won't be possible to write:

let simple = (v) => Promise.({
  return(v);
})

Maybe instead we should ask to put return into Promise directly?

(btw. In my prev comment I forgot to wrap the last expression of getJSON into Promise so that example fixes it)

The word do does imply side effect imho. It might not be the right word for things like async/option monads.

I think do is generic enough:

option — do means "do try to unwrap"
async — do means "do wait for resolution"

Now you can say that such "genericness" is not a good thing. But now I think we should've been arguing whether we need a generic monadic syntax or async-centered syntax.

Anyway the separate keyword do (or another) on RHS looks to me like it's a better solution than a monadiclet keyword (let! or lat or let%bind) on LHS:

monadic let keyword on LHS maybe seen as it introduces bindings in some different way (related to scope), while having a monadic keyword on RHS hints that it is about binding to a value, not the nature of binding a name.
I don't like punctuation in let! or let%bind. lat is ok but it suites async use case only, I think.

I think it would be easy to explain do for async computations for people with JS background: "it's like await but called do". Then later we can introduce the generic nature of do for other monads (option, result, ...).

The syntax with local module open already looks very similar to async/await functions in JS:

let getJSON = (url) => Promise.({
  let resp = do fetch(url);
  let data = do resp.json();
  return(Yojson.Safe.parse(data));
});

If we can somehow make local module open syntax "less noisy", then it gets even better:

let getJSON = (url) => Promise.{
  let resp = do fetch(url);
  let data = do resp.json();
  return(Yojson.Safe.parse(data));
};

Lupus commented 6 years ago

What about discussion in BuckleScript/bucklescript#1326? Citing proposed solution below:

let [@bs] v0  = promise0 in  (* default to Promise.bind *)
let [@bs] v1 = promise1 in 
let [@bs Option] v2 = optionl2 in  (* now default to Option.bind *)
let [@bs ] v3 = optional3 in 
let [@bs Promise] v4 = promise4 in (* back to promise *)
let [@bs error] v5 = error_handling in  (* here bind to Promise.catch *)
...

ncthbrt commented 6 years ago

If I could throw my 2c in, as someone relatively new to the language, I dislike the use of[@bs]. It is quite confusing. For someone who is simply trying to learn a new language and its paradigms, it greatly increases the cognitive load. It pulls in buckle script and ocaml into the discussion, and generally feels like a leaky abstraction. If the goal is to make reason an approachable language, especially to those coming from JS-land, I really think [@bs] should be avoided, especially for commonly used constructs like promises.

I really like @andreypopp 's "do" proposal however.

let-def commented 6 years ago

I like @andreypopp 's "do" proposal too.

Getting local module open to be less noisy shouldn't be a problem. It is not really clear to me what the scope would be for the different computations though: which opens are introduced by the sugar? Is it only about introducing the appropriate Let_syntax.bind and leaving the scope unchanged?

let-def commented 6 years ago

Note: %bind let x = ...; is already supported but the printing is not pretty. https://github.com/facebook/reason/pull/1703 tries to fix that.

hcarty commented 6 years ago

For something like the do proposal would something other than local open be possible? Some kind of annotation indicating what module should be used during the syntax transformation. This would help avoid unintentionally pulling values in. Something along the lines of (building on the example @andreypopp wrote):

module Promise = {
  module Let_syntax = {
    let bind = (~f, promise) =>
      Js.Promise.then_(f, promise);

    module Open_in_body = {
      let return = (v) =>
        Js.Promise.resolve(v)
    }
  };
};

let getJSON = (url) => {
  /* Some syntax to say that we want to use Promise for any "do" syntax
      without opening the module */
  do with Promise;
  /* Or some ppx-like [%do Promise]; */
  let resp = do fetch(url);
  let data = do resp.json();
  return(Yojson.Safe.parse(data));
};

andreypopp commented 6 years ago

This would help avoid unintentionally pulling values in.

Maybe we can solve that by educating people not to have unnecessary values inside modules which provide Let_syntax?

module Promise = {
  module Do = {
    module Let_syntax = {
      let bind = (~f, promise) =>
        Js.Promise.then_(f, promise);
      }
    };
  };

  let someOtherBinding = ...
};

let getJSON = (url) => Promise.Do({
  let resp = do fetch(url);
  let data = do resp.json();
  return(Yojson.Safe.parse(data));
});

That means we won't need to introduce another syntax construct. Also it leaves power users with the ability to define some utilities which can be used along with do-syntax w/o another open.

hcarty commented 6 years ago

We could minimize the syntax impact by using something closer to existing syntax like let do = Promise; One benefit of special syntax is that it makes it somewhat easier to find where the currently in-scope do handling comes from. If it happens via open, local or otherwise, then the source of Let_syntax is harder to find.

jordwalke commented 6 years ago

@andreypopp @let-def What would you imagine we do for constructs like switch?

If your proposal draws attention to the similarity between async/await, then I would imagine you'd have the following special cases for switch/if:

do M;
bind (M, ~f:(() => E))

let P = do M;
bind (M, ~f:(P => E))

switch (do M) {
  | P1 => E1
};
bind (M, ~f:(fun P1 => E1));

if (do M) {E1} else {E2};
bind(M, ~f:(fun true => E1 | false => E2))

The thing I like about putting the keyword on the right hand side is that it's then very clear exactly which value is being "awaited on". The LHS approach does hint at what's actually going on with the heavy rewriting of let bindings - however people don't care about the details - they only care about their mental model of how async values work.

I'm still not sure about the do keyword - it really does seem to imply effects. I am happy with a general purpose syntax transform for all monadic abstractions, but I wish the keyword conveyed that generality. Obscure symbols instead of do could help keep it general. (I realize that these monadic binders really are imperative under the hood, but to the end user, they often allow a transformation into a more declarative style).
I don't know if implicit return should be used. I just don't have enough experience with Lwt to consider if it's a good idea. I wish others with more experience would weigh in.
It's unfortunate to have to repeat the do keyword for and bindings.
With the suggestions for let x = do expr and if(do expr) I mentioned, people will expect that they will be able to "await" on any expression inline. Would you propose supporting do expr everywhere, and not merely special casing switch/if? That might be more involved. The left-hand-side syntax circumvents that problem.

hcarty commented 6 years ago

Implicit return seems like a bad idea. For example, the value may already be wrapped appropriately so adding return would add an extra layer to the result.

jaredly commented 6 years ago

For the record, here's the blog post I wrote on this subject recently: https://jaredforsyth.com/2017/12/30/building-async-await-in-reason/ (and accompanying ppx https://github.com/jaredly/reason_async) It touches on many of the topics discussed here.

(I'm also rather against a do notation, I think it's far too difficult for newcomers to understand. I very much prefer verbosity in this case)

jordwalke commented 6 years ago

@jaredly What are your thoughts on async/await (in JS?)

andreypopp commented 6 years ago

I'm now enjoying Reason with let%ext syntax. I'm thinking though if we can choose the more convenient char to type instead of % — maybe let/bind?

let-def commented 6 years ago

I like let/bind (as an alternative to https://github.com/facebook/reason/pull/1735)

andreypopp commented 6 years ago

@let-def #1735 looks amazing! Even better than let/bind. Would that syntax work if Reason decides to drop parens in if, switch and others?

jordwalke commented 6 years ago

@andreypopp I do not believe it will allow us to make parens optional for if/switch.

adrianmcli commented 6 years ago

Is there any progress on this so far?

I also think that do notation and implicit returns might be a bad idea here.

@jaredly's blog post on the matter is awesome, but the illustrated implicit return requires introducing awaitWrap and sacrifices the (rather helpful) explicitness of Promise.resolve or an otherwise explicit keyword.

I can foresee the difficulty in explaining awaitWrap to beginners who are not familiar with FP type signatures and knowledge of map and bind. It's much easier for their mental models if you just tell them "you're doing async stuff, so make sure to use this keyword to return the values".

wiltonlazary commented 6 years ago

Coroutines like Kotlin: Implements coroutines and let the rest builds around then, Kotlin coroutines infrastructure is amazing and lets you keep track of context.

danny-andrews-snap commented 6 years ago

I've never been a fan of the specialized syntax for promises in JS. Something like generators (coroutines) or do-notation are much more general and flexible. We'd do newcomers a disservice by assuming they can't learn/understand them.

wiltonlazary commented 6 years ago

@danny-andrews-snap have a look on my DSL proposal, this will make others additions like coroutines to don' t need a new syntax change to look like normal language syntax, https://github.com/facebook/reason/issues/1982

jaredly commented 6 years ago

@danny-andrews-snap I know there can be elegance to treating all monads the same, but I think async/await and optional chaining are pretty compelling specializations (in javascript and rust, respectively). to my mind, it's not a matter of "this is too hard for newcomers to learn", and more "what will make things easiest to read & maintain for everyone". I'm not convinced that do-notation on its own accomplishes that.

wiltonlazary commented 6 years ago

My Continuation-passing style PPX is almost ready with suport for basic syntax tree constructions, async/await and any suspendable function can be implemented on top of it, no need to create new keywords or change the parser, its uses only one attribute at function declaration [@cps] and you can be monadic and await on a promise if you wish, but it is not necessary if you uses the continuation transparent style.

texastoland commented 6 years ago

Would algebraic effects provide a primitive for this kind of thing? Or would it not translate to JavaScript? Or is that too far in the future like implicits?

graforlock commented 6 years ago

What is wrong with how F# solves async/await for example?

https://docs.microsoft.com/en-us/dotnet/fsharp/tutorials/asynchronous-and-concurrent-programming/async

Like:

let fetchHtmlAsync = url => 
    async {
        let uri = Uri(url);
        use webClient = new WebClient();
        let! html = webClient.AsyncDownloadString(uri);
        html
    }

That is beautiful, everyone agrees?

wiltonlazary commented 6 years ago

take a look on, https://github.com/wiltonlazary/cps_ppx

graforlock commented 6 years ago

@wiltonlazary I think if you documented it with extended README for simple API usage that would help

wiltonlazary commented 6 years ago

https://github.com/wiltonlazary/cps_ppx/blob/master/tests/src/js/IndexJs.re

wiltonlazary commented 6 years ago

I think that js only people dont know well the concept of coroutines that async computation can be built around

graforlock commented 6 years ago

I think it's actually quite well known in the JS world, for example redux-saga is like CSP with a single channel, and Node has a very popular library co. Mapping to an implementation to a generator runtime (like the implementation of babel's one) would also work with not much effort. The question is Reason syntax.

wiltonlazary commented 6 years ago

ReasonML is going on direction of monadic computation for async, i dont think that folks here are very inclined to coroutines or the https://fsharpforfunandprofit.com/series/computation-expressions.html

texastoland commented 6 years ago

@wiltonlazary informative link. The discussion is a bit disconnected but it does seem to be leaning toward generalized CPS. I personally admire F#. let! however is explicitly monadic.

Being inspired by Haskell do notation is also its shortcoming. It restricts you to a single Bind per context. By comparison ppx_let can intermingle bind, map, and both.

both is particularly interesting because it's for concurrent computations. Haskell uses apply for this. both in Haskell would be both = liftA2 (,) or simply "apply each then tuple". Likewise Haskell provides sugar for apply with a language extension called ApplicativeDo.

I started by saying the discussion is leaning towards generalized CPS. Both let! and ppx_let expose a simultaneously principled yet limiting interface. The idea is to expose 1 normal function for each let instead of 1 interface for all lets in scope.

PS neat PPX. I think the syntax could be less verbose. A better forum for feedback would be reasonml.chat.

graforlock commented 6 years ago

Computation expressions would be great as they are incredibly clean in F#. Also, try/catch in such implementation becomes much easier to reason about.

wiltonlazary commented 6 years ago

Try to not reinvent the wheel, my life is much easy now thanks to the fantastic implementation of coroutine concepts as a flexible framework by https://kotlinlang.org/docs/reference/coroutines.html

kuwze commented 6 years ago

I have no clue if I am actually contributing, but I found this recent paper with the title Try/Catch and Async/Await are just a specialized form of Algebraic Effects! on Reddit and thought it might be relevant.

OvermindDL1 commented 6 years ago

I have no clue if I am actually contributing, but I found this recent paper with the title Try/Catch and Async/Await are just a specialized form of Algebraic Effects! on Reddit and thought it might be relevant.

Yep, that's a pretty standard example of Algebraic Effects actually (as are faking mutable variables in an entirely immutable world).

OCaml has a HUGE side-developed system that includes one of the best Algebraic Effects systems I've seen short of having unbounded continuations and I hope to see it mainlined someday, but it seems slightly stalled (it would be SO useful though, especially combined with the multi-core changes).

But Reason would not be able to emulate Algebraic Effects at this time, however the OCaml work on it would no doubt mind another programmer working on it, thus meaning Reason would then just transparently support it as well. :-)

yunti commented 5 years ago

How will the recent merge of monadic and applicative let bindings in Ocaml affect this proposal? Can we just follow their lead?

reasonml / reason