Async generators are implemented in a way that involves a Jenga tower of state machines, wrappers and special exceptions.
I won't attempt to describe it further.
Instead I'll explain how they should be implemented.
Async generators extend the three-way exits of generators to a four-way exit.
Async generators may exit by yielding, awaiting, returning or raising.
The first thing to note is that being awaited is implemented the same as yield. This is fine for generators (which can yield, but cannot be awaited), or coroutines (which can be awaited, but not yield).
To avoid having to change that, we treat yielding in an async generator as a special operation, which I will call async yield.
Currently async yield is implemented in the bytecode as
First let's merge those into a single instruction: ASYNC_YIELD_VALUE.
In much the same way that YIELD_VALUE and RETURN_VALUE jump to different location in the caller, ASYNC_YIELD_VALUE will need to jump to yet another location. There is space for an 8 bit "async yield" offset to be added to the frame. Since the yield from loop is small, 8 bits will be sufficient offset from the yield target.
So we have:
return jumps to frame->prev_instr + frame->return_offset
yield jumps to frame->prev_instr
async yield jumps to frame->prev_instr + frame->async_yield_offset
To take advantage of this, we will also need to change the bytecode of the async for loop code to contain all the necessary offsets.
Currently the inner SEND has the async_yield_offset, but the yield_offset is implicit in the exception handling table.
Async generators are implemented in a way that involves a Jenga tower of state machines, wrappers and special exceptions. I won't attempt to describe it further.
Instead I'll explain how they should be implemented.
Anyone interested in this should read https://github.com/faster-cpython/ideas/issues/448 first. https://github.com/faster-cpython/ideas/issues/448 explains how we extend the simple two-way exit of return and raise, to include yield.
Async generators extend the three-way exits of generators to a four-way exit. Async generators may exit by yielding, awaiting, returning or raising.
The first thing to note is that being awaited is implemented the same as
yield. This is fine for generators (which can yield, but cannot be awaited), or coroutines (which can be awaited, but not yield). To avoid having to change that, we treat yielding in an async generator as a special operation, which I will callasync yield. Currentlyasync yieldis implemented in the bytecode asFirst let's merge those into a single instruction:
ASYNC_YIELD_VALUE.In much the same way that
YIELD_VALUEandRETURN_VALUEjump to different location in the caller,ASYNC_YIELD_VALUEwill need to jump to yet another location. There is space for an 8 bit "async yield" offset to be added to the frame. Since theyield fromloop is small, 8 bits will be sufficient offset from theyieldtarget.So we have:
returnjumps toframe->prev_instr + frame->return_offsetyieldjumps toframe->prev_instrasync yieldjumps toframe->prev_instr + frame->async_yield_offsetTo take advantage of this, we will also need to change the bytecode of the
async forloop code to contain all the necessary offsets. Currently the innerSENDhas theasync_yield_offset, but theyield_offsetis implicit in the exception handling table.