kedro-org / kedro

Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
https://kedro.org
Apache License 2.0
9.83k stars 895 forks source link

User report Hook with keyword argument does not work #4098

Open noklam opened 3 weeks ago

noklam commented 3 weeks ago

Description

An user struggle with using before_pipeline_run hook, upon investigation I think the root cause is in pluggy. Slack archive: https://linen-slack.kedro.org/t/22922543/hi-all-i-have-a-profiling-hook-that-uses-the-before-pipeline#42b185d3-2eca-445b-80f7-514b35bd6d12

class DummyHooks:
    @hook_impl
    def before_pipeline_run(self, run_params: Dict[str, Any] = dict()):
        print(run_params)

The user use the above hook and find run_params result in {}, this has nothing to do with the default value. If we change run_params=123, the hook will print 123 instead. In pluggy documentation it never use keyword arguments, but also didn't explicitly said this is not allowed.

I checked the implementation and don't immediate understand why keywords is not possible, so it could be either a bug or intentional. I follow up in this issue: Github: https://github.com/pytest-dev/pluggy/issues/522

noklam commented 3 weeks ago

I put this as a community ticket as there is no immediate action we can do. If pluggy confirm this is expected, maybe we can add some documents on our side to make this obvious.