Refactor Kedro-Viz FAST API app creation

ravi-kumar-pilla commented 1 month ago

Description

On Kedro-Viz we have a FAST API app in the backend and we use uvicorn web server to host the app. We register our routers and also install corresponding dependencies.

We include 2 routers before we start the app -

Register a REST API router and the required dependencies
Register a graphQL router and the required dependencies

We also do some expensive operations like SQLiteStore initialization, import modules like NetworkX, Azure related imports for DeployerFactory.

These can slow down the app and increase the start time.

Context

We need to refactor the app creation phase and also defer unwanted imports to improve the app start time.

Possible Implementation

To improve kedro viz run time after run command is reached, we can explore and experiment with changes I started here based on the flame graph

Some of the observations made while working on improving performance for Kedro-Viz -

Module imports related to experiment tracking, strawberry graphQL (2-3sec) - We need to see if we can delay graphql call until someone clicks on ET, remove getVersions call (a middleware which ingests graphQL api app once a request for graphQL is made)
Separate api app (main api + additional api) - This will help remove dependencies involved for other apis
Module imports related to deployer factory (2-3sec) - Deferring deployer factory imports until a /deploy call is made
Separate responses module and try to get the main route working quickly - This needs refactoring and breaking down our responses module.
NetworkX takes time for import - Deferring the networkX import until used
SQLiteStore initialization also takes time (reason for configure and bootstrap project taking time) - Need to find a way to defer store initialization (likely a North star)
Apart from all the above improvements, uvicorn.run takes ~10sec to make the app available. Need to experiment by making a lighter app, reducing api routes and dependencies.

Checklist

[x] Include labels so that we can categorise your feature request

astrojuanlu commented 1 month ago

Is it a good moment to ask whether it still makes sense to have a GraphQL and a REST API? :)

ravi-kumar-pilla commented 1 month ago

Is it a good moment to ask whether it still makes sense to have a GraphQL and a REST API? :)

Are you suggesting to move completely for GraphQL ? :D

I am not sure if there is an active discussion to remove Experiment Tracking completely. If so, we can definitely scrape the SQLite store and GraphQL. That would be a refactor PR to remove ET completely (remove FE too ?) .

astrojuanlu commented 1 month ago

Thanks @ravi-kumar-pilla, that's useful. So GraphQL exists solely for the Experiment Tracking part these days?

There's an active proposal yes https://github.com/kedro-org/kedro-viz/discussions/1831

My point is that, if we're to embark on a big refactor, the less code you have the fewer things you have to refactor 🙃

At the very least we could think of making it optional. Let's discuss that in https://github.com/kedro-org/kedro-viz/issues/2079

kedro-org / kedro-viz