Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
There are several features across the Kedro organisation that could benefit from manual testing on large projects to evaluate performance. The proposal is to create several kedro projects of varying size that can be used to test and experiment with.
The example projects don't have to be "legit" as in tackling real data science problems. It can be toy examples. The main point here is creating projects with scale.
This could be particularly useful for testing Viz features, CC @rashidakanchwala, @NeroOkwa
The expensive operation before starting the viz server is loading the data from the Kedro session (possibly related to Lazy Loading of Catalog Items kedro#2829 ?)
Most of the time taken to load the data is from catalog and pipelines_dict resolution, which worsens as the pipeline count increases
Since this is in sprint already, I update the title so this will be the ticket for pipeline stress testing. I will create separate tickets for the component part.
Description
https://github.com/kedro-org/kedro/issues/3957 needs to be done first as pre-work.
There are several features across the Kedro organisation that could benefit from manual testing on large projects to evaluate performance. The proposal is to create several kedro projects of varying size that can be used to test and experiment with.
The example projects don't have to be "legit" as in tackling real data science problems. It can be toy examples. The main point here is creating projects with scale.
This could be particularly useful for testing Viz features, CC @rashidakanchwala, @NeroOkwa
Context
The Kedro-Viz team carried out a performance analysis using an internal QB pipeline, with preliminary results shown here https://github.com/kedro-org/kedro-viz/issues/1064
(from https://github.com/kedro-org/kedro-viz/issues/1064#issuecomment-2100489744, summary of internal report).
There is preliminary evidence that the Kedro Framework CLI is a bottleneck for Kedro Viz.
This is on top of the already existing evidence that Kedro takes a lot of time to load even for trivial commands or almost empty projects #1476
We noted that there are several factors that make a pipeline "complex":
In I expanded on @AhdraMeraliQB's original proposal and suggested that we create a family of pipelines, comprising
Comes from https://github.com/kedro-org/kedro/discussions/3790
Since this is in sprint already, I update the title so this will be the ticket for pipeline stress testing. I will create separate tickets for the component part.