Closed achimnol closed 2 years ago
/cc @1st1 Yuri, you might consider this mode also when designing the asyncio TaskGroup class.
To aggregate the results of belonging tasks, we could use a separate async-generator channel to subscribe the results/exceptions of tasks finished through time, though this feature would be completely optional as the main use case for PersistentTaskgroup
would be server applications where most tasks would return just None
while serving their results to some I/O channels such as client connections.
Let's add a persistent task group in addition to the current
aiotools.TaskGroup
.While existing
TaskGroup
targets to group and ensure termination of a set of tasks with a short period of lifetime (e.g., a single operation that is composed of many async tasks),PersistentTaskGroup
targets to group and ensure termination of a continuously changing set of tasks that are created and terminate through a very long time (e.g., the entire process lifecycle), with proper cancellation and exception handling upon shutdown.One key difference to
TaskGroup
is that it has its own fallback exception handler because gathering the results (and exceptions) of all tasks at once at the point of "exit" asMultiError
is meaningless due to sparsity across time. Custom fallback exception handlers could be implemented using decorators (withfunctools.wraps()
to keep the function signature consistent with the original coroutine).To automatically detach terminated tasks from the tracking set, we could use
weakref.WeakSet
.In fact, the above pattern is widely used in most Backend.AI components, and adding
PersistentTaskGroup
will reduce boilerplates for Backend.AI.