search

kedro-org / kedro-viz

Visualise your Kedro data and machine-learning pipelines and track your experiments.
https://demo.kedro.org
Apache License 2.0
672 stars 110 forks source link

Enhancing Kedro-Viz Performance with Lazy Loading #1806

Open rashidakanchwala opened 6 months ago

rashidakanchwala commented 6 months ago

Description

The current loading strategy for Kedro-Viz can lead to significant delays. It requires all data, including session information like the catalog and session_store for experiment tracking, to be fully loaded before the visualization is rendered. This process can slow down the display of the Kedro-Viz flowchart, which only needs the quicker-to-load pipeline_data for initial rendering.

To improve the performance, we need to -

  1. Refactor Kedro-Viz CLI - https://github.com/kedro-org/kedro-viz/pull/1920
  2. Refactor Kedro-Viz FAST API app creation - https://github.com/kedro-org/kedro-viz/issues/2061
  3. Asynchronously load pipeline data - I tested this while profiling work and created a test branch with some changes in server.py - https://github.com/kedro-org/kedro-viz/blob/test/large-pipeline-profiling/package/kedro_viz/server.py . I did not see a huge improvement in load time, but we can take this up as a minor improvement. I will create a ticket if needed. cc: @rashidakanchwala

Testing the performance improvements -

  1. Test against a massive pipeline once created - https://github.com/kedro-org/kedro-viz/issues/1064
  2. Test against a complex pipeline once created - https://github.com/kedro-org/kedro/issues/3866

Checklist