As a Kedro user I have always wanted to be able to define modular pipelines in a config file.
Context
I believe that doing so will reduce the likelihood of a user inadvertently impacting a pipeline other than the one intended when making changes to pipeline_registry.
Possible Implementation
I created PR #3904 in the Kedro core repository and @datajoely mentioned it might make more sense to make this a plugin rather than part of Kedro Core, but I'm unsure which directory it would belong in.
This is just one idea on how the Developer Experience could be:
$ kedro new ... && cd my_project && uv venv && source .venv/bin/activate # assume (.venv) in all prompts
$ uv pip install -r requirements.txt
$ uv pip install kedro-yaml-pipelines # Your plugin
$ kedro yaml-pipeline create data_processing # Not particularly beautiful, open to ideas here
$ tree src/my_project/pipelines
src/my_project/pipelines/
├── __init__.py
├── data_processing
│ ├── __init__.py
│ ├── nodes.py
│ └── pipeline.yaml # <------- The YAML definition!
$ # Or, alternatively, YAML pipelines are defined in a central location?
$ tree src/my_project
src/my_project/
├── __init__.py
├── __main__.py
├── hooks.py
├── pipeline_registry.py
├── pipelines
│ ├── __init__.py
│ ├── pipelines.yaml # <---- All pipelines are defined here?
│ ├── data_processing.py # <---- Maybe even no need for `nodes.py`?
$ # Make edits to nodes, YAML definition
$ kedro run
...
# Everything works as usual!
Now, the only blocker I see off the top of my head is the pipeline_registry.py. Maybe you could tell the user to slightly modify it as follows:
from kedro.framework.project import find_pipelines
from kedro.pipeline import Pipeline
+from kedro_yaml_pipelines.registry import find_pipelines as find_yaml_pipelines
+
def register_pipelines() -> Dict[str, Pipeline]:
"""Register the project's pipelines.
@@ -12,5 +14,6 @@ def register_pipelines() -> Dict[str, Pipeline]:
A mapping from pipeline names to ``Pipeline`` objects.
"""
pipelines = find_pipelines()
+ pipelines += find_yaml_pipelines()
pipelines["__default__"] = sum(pipelines.values())
return pipelines
and hopefully this should be it.
Of course there are lots of variations for this. The key points are
This is your plugin. So feel free to tailor the DX to your needs. Don't let us dictate how it should be - my suggestions above are just suggestions.
We are happy to guide you on how to create such a plugin. Whether or not that becomes official is another story - but if we see enough traction, I think we should seriously consider it!
astrojuanlu
commented
3 weeks ago 
Description
As a Kedro user I have always wanted to be able to define modular pipelines in a config file.
Context
I believe that doing so will reduce the likelihood of a user inadvertently impacting a pipeline other than the one intended when making changes to
pipeline_registry.Possible Implementation
I created PR #3904 in the Kedro core repository and @datajoely mentioned it might make more sense to make this a plugin rather than part of Kedro Core, but I'm unsure which directory it would belong in.
Possible Alternatives
@datajoely mentioned a possible alternative here