search

kedro-org / kedro-devrel

Kedro developer relations team use this for content creation ideation and execution
Apache License 2.0
0 stars 3 forks source link

"Hamilton vs Kedro" plus "Hamilton and Kedro" #48

Closed stichbury closed 1 year ago

stichbury commented 1 year ago

How the two compare. How the two complement each other.

Based on some discussion following this discussion on hacker news and some ongoing discussion over on the Hamilton Slack

We used Kedro at MoovAI. The standardized structure is reaaally valuable in consulting where team members change over the course of a project! The folks at potloc like it a lot and presented it at the most recent Montréal MLOps community event! While using Kedro, I wanted to create modular functional code for data transformation, but creating a node for each function would require me to specify input-output for each node. In addition, if the output of these nodes would be pandas Series, I would have to assemble them manually at intermediary steps. That's when I learned about Hamilton, which exactly met my needs for quick iterations of data processing pipelines with little/no boilerplate. I ended up calling Hamilton within a single Kedro node! (similar to Metaflow+Hamilton) I think one of the main appeal of Kedro for orgs is the visualization tool that encompasses functions, data, code, experiments, etc. (+MLFlow and Airflow plugins). Integrating Kedro-viz with other DAG tools could be very exciting for users to have full visibility of their ETL pipeline. For example, at the MLOps meetup, someone asked if it would be possible to plug their Airflow ETL (upstream of the data science Kedro project) into the tool!

astrojuanlu commented 1 year ago

Shall we close this in favor of kedro-org/kedro#3012?