It is beneficial to minimize the number of environments used by PTransforms of a pipeline. For example,
We want to to reduce resource overhead due to additional environments running in workers
We want to eliminate serialization just to send data across environments.
While exact process for merging environment will depend on the type of environment, language etc. it will be useful to introduce a new merged environment type that refer to a list of sub-environments and can be addressable by any of the environment IDs of the sub-environments. This way the runner could simply startup this one merged environment and send bundles intended for all sub-environments to the merged environment.
Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
tvalentyn
commented
9 months ago
What would you like to happen?
It is beneficial to minimize the number of environments used by PTransforms of a pipeline. For example,
While exact process for merging environment will depend on the type of environment, language etc. it will be useful to introduce a new merged environment type that refer to a list of sub-environments and can be addressable by any of the environment IDs of the sub-environments. This way the runner could simply startup this one merged environment and send bundles intended for all sub-environments to the merged environment.
Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
Issue Components