search

kedro-org / kedro

Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
https://kedro.org
Apache License 2.0
10.01k stars 905 forks source link

Add support for multi-node for `%load_node` or `%load_pipeline` #4170

Open noklam opened 2 months ago

noklam commented 2 months ago

Description

%load_node assume error comes from the node but this is not always the case.

This happens more commonly in Data Engineering pipeline, where you apply a series of transformation, aggregation on a set of table and pass it to next node. For example, you may get a "Column not found" error. loading the node that thrown out an error is only the first step to inspect the data, but there are still couple of manual steps to figure out where is the source of error.

The process roughly work as a binary search of the upstream nodes.

Context

This augment the existing debugging feature of Kedro and making this much easier for DS & DE

Runner is an abstraction that is powerful but not beginner friendly, bring the execution explicitly into a notebook cell is helpful

It's not a trivial task to figure out the correct execution order from a Kedro pipeline to a imperative manner (i.e. cells run sequentially in a notebook). The abstraction is a distraction mostly during debugging.

Possible Implementation

Limitation: Creating multiple cells is not easy, I tried in %load_node the first time but settle with the current solution because IPython do have limitations. We may be able to do this in Jupyter Notebook (not VSCode notebook) because there are better support

Possible Alternatives

dundermain commented 10 hours ago

Hey @merelcht and @noklam , I would like to work on this issue. Let me know if that is okay with you.

noklam commented 4 hours ago

@dundermain awesome, we haven't started with this ticket, but if this is something you would like to take a stab, go for it.