kedro-org / kedro-plugins

First-party plugins maintained by the Kedro team.
Apache License 2.0
92 stars 89 forks source link

Performance profiling on `kedro new` with starter and without starter #120

Closed astrojuanlu closed 10 months ago

astrojuanlu commented 1 year ago

When the telemetry is enabled and the Internet connection is choppy or poor, kedro new takes a lot to run.

It could be because of the telemetry timeout:

https://github.com/kedro-org/kedro-plugins/blob/7fc511f2013002d2d85b89014f49384f735e6f76/kedro-telemetry/kedro_telemetry/plugin.py#L204

Or it could be because of cloning the starter itself. This is related to https://github.com/kedro-org/kedro/issues/1476

astrojuanlu commented 1 year ago

Example from yours truly:

% /usr/bin/time -p kedro new --starter=standalone-datacatalog --config=kedro.yaml

The project name 'kedro-polars' has been applied to: 
- The project title in /Users/juan_cano/Projects/QuantumBlack Labs/talk-kedro-polars/kedro-polars/README.md 
- The folder created for your project in /Users/juan_cano/Projects/QuantumBlack Labs/talk-kedro-polars/kedro-polars 
- The project's python package in /Users/juan_cano/Projects/QuantumBlack Labs/talk-kedro-polars/kedro-polars/src/kedro_polars

A best-practice setup includes initialising git and creating a virtual environment before running 'pip install -r src/requirements.txt' to install project-specific dependencies. Refer to the Kedro documentation: https://kedro.readthedocs.io/

Change directory to the project generated in /Users/juan_cano/Projects/QuantumBlack Labs/talk-kedro-polars/kedro-polars by entering 'cd /Users/juan_cano/Projects/QuantumBlack Labs/talk-kedro-polars/kedro-polars'
real 24.91
user 1.43
sys 1.03

(almost 25 seconds for a kedro new)

ankatiyar commented 1 year ago

Suggestion: Making kedro-telemetry use after_command_run hook instead of before_command_run might solve this problem!

astrojuanlu commented 1 year ago

Thanks @ankatiyar! I think when I opened this issue I had the impression that this was because of telemetry, but actually it could be cloning the starter. I'm renaming this to clarify.

astrojuanlu commented 1 year ago

Actually, maybe we should close it in favour of https://github.com/kedro-org/kedro/issues/1476

merelcht commented 10 months ago

Closing in favour of https://github.com/kedro-org/kedro/issues/1476