kedro-org / kedro

Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
https://kedro.org
Apache License 2.0
9.53k stars 877 forks source link

Spike: Provide a way for plugins to have runtime configuration and extend CLI arguments #2866

Open noklam opened 12 months ago

noklam commented 12 months ago

Description

settings.py is the place that where you can disable/enable plugins, however it's not a good place to put runtime argument in.

Consider that I have a plugin which utilize hooks to do runtime validation. The ideal way of doing so is kedro run --my_plugin_check=True. However, it's not possible to directly override the kedro run command.

Possible Implementations/Issues

  1. Override cli.py - this is documented badly - It's probably not a good choice because you need to do this per project, plugin have no control over it.
  2. Override kedro run - If plugins are allowed to override/extend kedro run (Is this even feasible?) - there are risks that argument will conflict with each other.

What are the workarounds?

Outcome of this Issue:

Context

Related to #2384 in terms of adding new arguments for kedro run, but may not be a complete replacement.

Possible Implementation

noklam commented 12 months ago

Potentially related to https://github.com/kedro-org/kedro-viz/issues/662 too