Define a policy for resuming leaf operations that avoids running continuations inline on context that triggered completion

[lewissbaker]

There are several data-structures that have async operations where a consumer that waits on something is un-blocked by some other operation on the data-structure.

For example:

• bounded_queue from P0260 has an async_pop() that returns a sender. If the queue was empty at the time the operation was started then it would be completed by the next call to try_push(), push() or async_push(). However, completing the async_pop() operation inline inside the call to push() could then potentially run an arbitrary amount and type of code inside the call to push(). Ideally, the call to push() would just enqueue the completion of async_pop() to a scheduler and then return immediately.
• async_scope from P3149 has a join() operation on the counting_scope object, but the join() operation generally completes as a side-effect of an operation-nested within the scope completing. e.g. triggered from the destructor of a future-sender or nest-operation-state. If the join() operation completes inline inside the destructor then it might call continuation code that runs an arbitrary amount of code inside the context of that destructor, delaying the execution of the continuation attached to the future-sender operation. Ideally, decrementing the last ref-count would just trigger scheduling an operation on to some scheduler that then invoked the completion-handler for the join() operation.

These are just two examples of this kind of "something happens that triggers completion of some waiting operation" situation - this situation will recur on almost all data-structures that have a "wait until something happens" async operation.

We need to come up with a strategy/pattern that we can apply to such data-structures to ensure that they have consistent behaviour that doesn't have the inline-completion footguns.

Several options to consider:

• require that the receiver connected to the waiting-op-sender has a scheduler and either have it complete inline or on the associated scheduler
• introduce a new algorithm that can be adapted over each leaf operation that has the operation either complete synchronously on the start() context, or otherwise schedules completion on a specified scheduler - similar to folly::coro::viaIfAsync().
• Just leave it as is and require users to manually apply completes_on() to the sender to force the completion to reschedule onto a provided scheduler

Note that this may tie in with the task design - the task coroutine type may implicitly apply such an algorithm to all co_await expressions within the coroutine.