search

kedro-org / kedro

Kedro is a toolbox for production-ready data science. It uses software engineering best practices to help you create data engineering and data science pipelines that are reproducible, maintainable, and modular.
https://kedro.org
Apache License 2.0
9.53k stars 877 forks source link

Request: kedro-kubeflow plugin #377

Closed Minyus closed 3 years ago

Minyus commented 4 years ago

Description

A plugin to productionize Kedro pipelines using Kubeflow.

Context

Besides Airflow, Kubeflow is becoming another powerful platform for production. Discussed at https://github.com/quantumblacklabs/kedro/issues/353 : @jaklan @kaemo @yetudada

Possible Implementation

Convert Kedro pipeline into Kubeflow pipeline. Reference: https://www.kubeflow.org/docs/pipelines/sdk/sdk-overview/

Possible Alternatives

Use kedro-argo package which converts Kedro pipelines to Argo (backend of Kubeflow) pipelines.

yetudada commented 4 years ago

As mentioned in #353 this will be delivered as part of our "Universal Deployment" epic.

yetudada commented 3 years ago

As mentioned in #353 we've created documentation on how to deploy Kedro on Kubeflow. We'll be tracking this page to see how much interest there is in this and then we'd build for it, if there's enough demand. There's also a guideline for Argo too. In the interim I'll close this ticket.