search

kedro-org / kedro

Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
https://kedro.org
Apache License 2.0
9.84k stars 893 forks source link

Performance analysis and smart re-running of pipelines #2057

Open merelcht opened 1 year ago

merelcht commented 1 year ago

Description

Kedro currently doesn't offer any options to analyse the performance of pipelines. Additionally, our users have flagged that they would like to be able to re-run only parts of their pipeline.

Implementation ideas

Questions

noklam commented 1 year ago

I think it's better to further split this into two issues, but I will leave my comments for both topics.

We have something like PipelineMonitoringHook in our docs. It requires some infrastructure and it's not easy to set up by regular users.

Pipeline Statistic

How would these pipeline stats be useful?

  1. Just having a summary of the statistic could be very useful to understanding the pipeline performance - instead of parsing the log by yourself
  2. In addition, we could add visualization on kedro-viz side to show where's the bottleneck and help user to optimize their pipeline
  3. The pipeline statistic could be useful for ParallelRunner or something similar. Currently, the workload is distributed naively, but not every node is equal.

Smarter way to re-run the pipeline

Similar to the Pipeline's run_only_missing, but a more sophisticated one. During development, it's common that you are working on one particular node and you just need to refresh one node(or a few dependent nodes). We can so some back-tracking.

Alternative

Currently, it requires users to figure out which nodes are not necessary, and do kedro run --from-nodes to skip unnecessary computation

Summary

One key realization of this change is that Run need to have memory. To optimize runtime performance, it needs to know how it is run previously. To re-run the pipeline in a smart way, it needs to know the previous run(s) and figure out what's the minimal computation.

noklam commented 1 year ago

Related Issue:

astrojuanlu commented 9 months ago

This is a very frequent question actually, will try to collect more evidence for it going forward.

There's different things when considering performance, namely (1) execution time, and (2) RAM usage. There are different tools for each of these purposes, so most likely we would need dedicated efforts.

I think execution time is probably the most urgent one. This is how I used pyinstrument https://github.com/kedro-org/kedro/issues/3033#issue-1895014637