A data pipeline orchestration library for rapid iterative development with automatic cache invalidation allowing users to focus writing their tasks in pandas, polars, sqlalchemy, ibis, and alike.
If someone likes to programmatically produce pipedag configuration, it should be possible to construct a PipedagConfig object from a dictionary. Ideally, it is possible to input PipedagConfig.raw_config into that constructor and yield exactly the same PipedagConfig.config_dict.
An extended version of implementing this feature would offer some programmatic calls which can produce config dictionaries which are good default configurations with very little code.
Hacky workaround code as long as this feature is not available:
If someone likes to programmatically produce pipedag configuration, it should be possible to construct a PipedagConfig object from a dictionary. Ideally, it is possible to input PipedagConfig.raw_config into that constructor and yield exactly the same PipedagConfig.config_dict.
An extended version of implementing this feature would offer some programmatic calls which can produce config dictionaries which are good default configurations with very little code.
Hacky workaround code as long as this feature is not available: