It could be useful to be able to export or import pipelines from a static file definition like .yaml for the sake of running larger experiments. One such example is running things with Hydra or when you have to run many experiments parallelized over a cluster.
This could be fairly straightforward with most pipeline setups that rely on libraries for their components and involve simple config and spaces.
Some clear problems with this of course are functional elements of a pipeline, for example the config_transform, which typically invoke user based transforms during the configuration of a pipeline, or custom components.
These problems could be possible to solve as hydra seems to do some import magic to make it happen, at the cost of less transparency. This also means exported pipelines will not be transferable from one environment to the next.
It could be useful to be able to export or import pipelines from a static file definition like
.yamlfor the sake of running larger experiments. One such example is running things with Hydra or when you have to run many experiments parallelized over a cluster.This could be fairly straightforward with most pipeline setups that rely on libraries for their components and involve simple
configandspaces.Some clear problems with this of course are functional elements of a pipeline, for example the
config_transform, which typically invoke user based transforms during the configuration of a pipeline, or custom components.These problems could be possible to solve as hydra seems to do some import magic to make it happen, at the cost of less transparency. This also means exported pipelines will not be transferable from one environment to the next.