TaskGroups straddling async context managers are not supported

[adriangb]

Because of the fundamental design decision of executors controlling execution of non-context manager dependencies and setup of context manager dependencies but not teardown of context manager dependencies (those get run when the scope is exited via an AsyncExitStack) it is not possible to have a TaskGroup (or anything making use of a CancelScope) straddle the yield in the context manager because the cancel scope would be exited in a different task than it was entered in!

Here's a simplified example of what's going on:

from contextlib import AsyncExitStack, asynccontextmanager
from typing import AsyncContextManager, AsyncIterator, Callable

import anyio

@asynccontextmanager
async def cm_with_cancel_scope() -> AsyncIterator[None]:
    with anyio.CancelScope(): 
        yield

async def run_setup_in_tg(stack: AsyncExitStack, cm: Callable[[], AsyncContextManager[None]]) -> None:
    # Task schedules it's own teardown, which happens outside of the task group we are currently running in
    await stack.enter_async_context(cm())

async def main() -> None:
    async with AsyncExitStack() as stack:  # inside container.enter_scope(
        async with anyio.create_task_group() as tg:  # inside ConcurrentAsyncExecutor.execute
            tg.start_soon(run_setup_in_tg, stack, cm_with_cancel_scope)

anyio.run(main)

Note that this does not impact:

• Using a TaskGroup that is completely contained within the startup or shutdown of the context manager
• Anything using AsyncExecutor (and not ConcurrentAsyncExecutor).

The only "solution" to this I can think of is to create the TaskGroup when entering an async scope:

async with container.enter_scope("app"):   # implicitly creates a TaskGroup and somehow passes it down into the executor
    ...

@graingert if you have any thoughts I would love your opinion on this 😄