Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
Add run-time checks for plugins. Related to #2685 , to lower the learning curve of creating plugin.
While I am developing https://github.com/Galileo-Galilei/kedro-pandera/pull/10 I found that I have a typo, it is hard to debug because pip install -e . would not update the entry_points automtically, I need to re-run pip instal -e .. Initially I thought I was not registering the hook correctly, turns out that it is because I use before_dataset_load instead of before_dataset_loaded, kedro should raise an error.
Context
Develop a new hook / plugin is hard. One of the common trap is that invalid hook fails silently and makes it hard to debug.
This should check multiple things
Both CLI/Kedro hooks should be validated
Hooks installed from plugins should be validated too
Description
Add run-time checks for plugins. Related to #2685 , to lower the learning curve of creating plugin.
While I am developing https://github.com/Galileo-Galilei/kedro-pandera/pull/10 I found that I have a typo, it is hard to debug because
pip install -e .
would not update theentry_points
automtically, I need to re-runpip instal -e .
. Initially I thought I was not registering the hook correctly, turns out that it is because I usebefore_dataset_load
instead ofbefore_dataset_loaded
,kedro
should raise an error.Context
Develop a new hook / plugin is hard. One of the common trap is that invalid hook fails silently and makes it hard to debug.
This should check multiple things
Possible Implementation
Leverage https://pluggy.readthedocs.io/en/stable/api_reference.html#pluggy.PluginManager.check_pending
2863 is a quick PoC to points out how it can be used.
When an invalid Hook is passed.
Running Kedro with
kedro run
with result in thisPossible Alternatives