Lazy load commands from plugins

ankatiyar commented 4 months ago

Description

Partly resolve #1476

As discussed in #1476, the initialisation of KedroCLI takes up a significant chunk of time. This is especially evident if you have a lot of kedro plugins installed in the environment as KedroCLI which is a CommandCollection, loads up all the commands from the plugins before the command is processed. This PR is to add a lazy way to load the commands from the plugins.

Development notes

Add a property called plugin_groups which returns a dict of entry_point_name to the EntryPoint object.
This group is passed to the super class CommandCollection which is a custom version of click.CommandCollection used by Kedro and defined in kedro/framework/cli/utils.py

Update to custom CommandCollection:

Add a new main function which receives the command to be run as args, eg [ "catalog", "list"] or ["airflow", "create"]
if the first element of the args is not in the command, then load commands from plugins
run the command by passing to super().main()

Loading of plugins:

I tried to implement a sort of "smart" loading where if the command arg, i.e. airflow or mlflow or viz partially matches the entry point names in the lazy_group dict keys, load the plugin, check if the command now exists and exit. Note: The entry point names are decided by the plugins, eg. kedro-airflow's ep name is airflow which fully matches the command name too but for kedro-viz and kedro-mlflow the entry point names don't match the commands. eg. Command: viz Entry point name: kedro-viz

Otherwise, load all plugins one by one, exit if the command exists after loading plugins.

TODO

Fix code coverage on unit tests, which is tricky since this code is only used when a plugin is installed but we don't have tests like these in our unit tests.

Developer Certificate of Origin

We need all contributions to comply with the Developer Certificate of Origin (DCO). All commits must be signed off by including a Signed-off-by line in the commit message. See our wiki for guidance.

If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.

Checklist

[ ] Read the contributing guidelines
[ ] Signed off each commit with a Developer Certificate of Origin (DCO)
[ ] Opened this PR as a 'Draft Pull Request' if it is work-in-progress
[ ] Updated the documentation to reflect the code changes
[ ] Added a description of this change in the RELEASE.md file
[ ] Added tests to cover my changes
[ ] Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team

astrojuanlu commented 4 months ago

To note, as @ankatiyar noted internally, "bad" lookups will take a lot of time: for example, kedro infoo (notice typo) took 17 seconds in my computer and tried to import kedro-viz, kedro-mlflow, and more.

Going forward, is there a smarter way we can declare new CLI commands, so that we can look them up in entry points without actually importing anything? (Feel free to open a new issue about this)