The previous implementation of sub plans API comprised of several modules which could be layered on top of each other. This made the code somewhat hard to follow, not all of the modules worked in all scenarios as some relied on an executor being a single process.
The v2 flattens all the modules into a single one and drastically reduces the amount of possible combinations. Sub plans are spawned in batches and then the parent polls until the sub plans finish.
It also includes support for limitting the number of sub-plans that may be running at a given time, without relying on the throttle limitter and thus should be safe in multi-process environments. Under the hood it dynamically alters the batch size to honor the concurrency limit.
TODO:
[ ] deprecate the old modules? It is probably still too early for this, let's not be hasty
adamruzicka
commented
1 year ago
The previous implementation of sub plans API comprised of several modules which could be layered on top of each other. This made the code somewhat hard to follow, not all of the modules worked in all scenarios as some relied on an executor being a single process.
The v2 flattens all the modules into a single one and drastically reduces the amount of possible combinations. Sub plans are spawned in batches and then the parent polls until the sub plans finish.
It also includes support for limitting the number of sub-plans that may be running at a given time, without relying on the throttle limitter and thus should be safe in multi-process environments. Under the hood it dynamically alters the batch size to honor the concurrency limit.
TODO:
deprecate the old modules?It is probably still too early for this, let's not be hasty