921kiyo
commented
4 years ago Hi @deepyaman Thank you for sharing your hooks! Regarding the contribution, Kedro only provides the hooks spec and doesn't have any concrete hooks implementation. Hooks implementations are something users would implement in their projects, and we would encourage people to publish their own hooks outside of Kedro hooks.
We are more than happy to list your hook plugins in our docs (see the examples for kedro-wing, and steel-toe)
Minyus
deepyaman
tsanikgr
Description
I'm always frustrated when I/O dominates compute.
For example, my pipeline takes 10 minutes to run, of which 9 minutes are spent writing to and reading back from S3.
Context
At QuantumBlack, it's most common to write intermediate datasets to disk. In fact, the Kedro data catalog very much facilitates this workflow. This also presents numerous advantages:
However, it's also extremely inefficient, especially when writing large datasets using slow mechanisms. On top of that, we most often expect reloaded data to be exactly equal to what was saved, save the case of transcoding and some terminal output formats (e.g. Excel, CSV).
Possible Implementation
https://github.com/deepyaman/hookshot/blob/develop/src/hookshot/hooks.py
Feel free to clone the repo and run the example. :)
At a high-level, the plugin aims to provide Unix
tee-like behavior to runners.Goals of this implementation:
Limitations:
SharedMemoryDataSetforParallelRunner. I would be happy to get some input from the experts here. :)I'm most interested in understanding what's the best way to contribute this. I think it makes sense as part of a new
kedro.extras.hookssubpackage. As part of Kedro, this functionality would continue to be supported through backend redesigns.