Add support for coroutines

[ahills]

Using the await special, a function becomes a coroutine suitable for use with Python's asyncio library.

CPython actually produces a YIELD_FROM opcode for await, which is mysteriously always (¿?) preceded by pushing a None value that is sent to the generator, coroutine, or iterator returned by GET_AWAITABLE or GET_YIELD_FROM_ITER[0]. With this patch, both the new await special and the yield-from special get a send: keyword argument that allows sending values other than None with YIELD_FROM. This doesn't appear to be defined behavior, so expect surprises.

[0] https://github.com/python/cpython/blob/b54a99d6432de93de85be2b42a63774f8b4581a0/Python/ceval.c#L2157

[obriencj]

This produces warnings under 3.5 and 3.6, and fails on 3.7 due to the name of some attributes. Apparently async and await are now reserved, so cannot be member, variable, nor parameter names.

[ahills]

After inspecting the behavior of CPython 3.6 & 3.8 more closely, I think this PR misses a major feature by eliding the async keyword: it's not possible to create a coroutine without using await, whereas CPython creates a coroutine from an async function that never awaits.

In other words, it's not possible with this implementation to create either x() or y() below:

async def x():
    return 1

async def y():
    yield 1

The former has the COROUTINE flag, and the latter has the ASYNC_GENERATOR flag (which I completely missed). An expression invoking a regular function will return a regular value instead of an awaitable, so can't be used with the await keyword.

I propose creating new coroutine definition facilities: (function …) → (async …)/(coroutine …), (def function …) → (def async …)/(def coroutine …), (defun …) → (defasync …)/(defcoro[utine] …)—any preference on names?

It might also be valuable to add an async decorator to flag a function or generator as async, i.e., (define coro (async (function …))) & (define agen (async (#gen …))). I'm not sure this would be as easy to implement elegantly, and is probably not as useful.

[obriencj]

I like “coroutine” or “async-function” both. Any ideas for lambda? “async-lambda” or “coro” ?

[obriencj]

I was also thinking that a macro like (declare-async) could be used to expressly set the coroutine flag in the current compiler. That would make it so defasync or coroutine could simply be macros instead of specials (ie. less python code, more sibilant code!)

[obriencj]

Was also thinking about the current implementation of let, maybe we should allow it to become a coroutine if await is used within it, and then check at the invocation of the let lambda to await rather than call it?

[obriencj]

(That can all come later, too, if you want to use this PR as-is)

[obriencj]

Did you want to merge this as-is and worry about other tweaks later?

[ahills]

No, I think the current design is wrong. It's important to be able to define coroutines without using the await keyword in them. I have a patch that adds the following macros & specials:

• declare-async (special) to be used anywhere within a function or generator body, marking it as asynchronous (COROUTINE or ASYNC_GENERATOR flags, respectively)
• coroutine (macro) for creating asynchronous function objects (coroutines), along with a define target (i.e., (def coroutine …), equivalent to async def in Python)
• coro (macro) for asynchronous lambdas (no Python equivalent)
• defcoro (macro) for global function definitions à la defun
• let& (macro) to make the let wrapper function asynchronous

The last one is a little strange, because it's not immediately obvious that the let implementation uses functions; if you're invoking it directly, you either need to be awaiting the result, or handing it off to one of the asyncio methods that handles awaitables. I am hoping that invoking the POSIX bourne shell syntax for asynchronous jobs indicates that you'll get something other than a normal synchronous return value.

Right now the patch's tests are incomplete, so I haven't pushed the changes up.