Closed noklam closed 6 months ago
How would you imagine this work?
feature branch: @datajoely https://github.com/kedro-org/kedro/compare/main...noklam/node-ordering-proposal
Don't focus on the API, it's hacky and there is many random Pipeline call in the current process so I have to patch that everywhere.
Demo Project:
comment or uncomment this line to play with it.
There are many way to build a nicer API, for example:
It's not too important to decide this now, it's more of a PoC to prove this is possible. Benefit of this is Kedro viz won't see this at all (I think)
We can also assume it always follow the order of declaration, or try to stick with it as much as possible. Those are just design decision so I will delay that discussion.
Okay so I understand what you're trying to do. But have some questions.
Why is asking users to pass dummy datasets between nodes insufficient? Few reasons:
- It contaminate the DAG flowchart (kedro-viz)
- It's hard to edit or read custom execution order, users will need to go through nodes and try to do that matching in their head or edit multiple file.
- It feels very hacky.
- It's annoying that you cannot make Kedro run sequentially (or at least follow the declaration order when it's possible)
Is a deterministic sort (something like a seed) order a more useful enhancement?
If you run with SequenetialRunner now, it's deterministic already https://github.com/kedro-org/kedro/pull/1604. Not sure if this is what you are talking about.
- This feels like something that should happen at a orchestor / namespace granularity level first, so would address some of the recommendations in #3094 first.
Which recommendation are you referring to? there are many mentioned.
@datajoely 2. is one the thing that annoys me before I joined thus the first thing I fixed :P (also requested by former colleague), CacheDataset is the other one that I still haven't fixed.
We'll get there 🚀
Description
Support custom / partial node order where user desired.
Background
Kedro offers 3 Runners out of the box (
SequentialRunner
,ParallelRunner
andThreadRunner
), there is anotherSoftFailRunner
which I have implemented and can be installed in https://pypi.org/project/kedro-softfail-runner/.In the past we have focus to make consistent support for runners and are reluctant to introduce feature parity. In fact, runners has been mostly unchanged for years. Improve resume pipeline suggestion for SequentialRunner introduce a concept similar to "Change Data Capture" (CDC), which fixed the broken suggestion and started to consider about persisted data and the closest checkpoint to recover a failed pipeline. This is the most obvious feature parity among runners as it only support
SequentialRunner
due to the non-deterministic nature of parallel computing.3026 attempts to fix this feature by considering "parameters" as persisted data.
Context
TBD
3713 - I started to think more about runner recently, and my feeling is that
SequentialRunner
is the most important one, Kedro doesn't play an important role in terms of helping user to get code executed in a parallel fashion. This problem is usually solved by the 3rd party library, for example, polars support multi-core computing out of the box.This proposal will only support
SequentialRunner
, which I am increasingly more comfortable with, details are discussed in #3713.Design
Toposort will give ONE feasible solution, while there can be multiple possible solutions. In some case:
Requirements:
Nice to have:
Feature:
Next step is providing an user friendly API, as this is likely still too low-level for the end user.
High level Proposal
Add new constrains to "node_dependencies" addition to the existing inputs/outputs pair.
Possible Implementation
Can't think of anything, thus dummy outputs has been the workaround for years.
Possible Alternatives
Current workaround involves dummy inputs outputs which become tedious quickly.