[Feature][Task Plugin] Add Kubeflow task plugin for MLOps scenario

jieguangzhou commented 2 years ago

Search before asking

[X] I had searched in the issues and found no similar feature requirement.

Description

The Kubeflow project is dedicated to making deployments of machine learning (ML) workflows on Kubernetes simple, portable and scalable. Our goal is not to recreate other services, but to provide a straightforward way to deploy best-of-breed open-source systems for ML to diverse infrastructures. Anywhere you are running Kubernetes, you should be able to run Kubeflow. ---- https://www.kubeflow.org

Kubeflow can give dolphinscheduler the ability to run machine learning tasks on K8S

part of #12230

Use case

No response

Related issues

No response

Are you willing to submit a PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

jieguangzhou commented 2 years ago

Has anyone used DolphinScheduler to schedule Kubeflow tasks？

github-actions[bot] commented 2 years ago

Thank you for your feedback, we have received your issue, Please wait patiently for a reply.

In order for us to understand your request as soon as possible, please provide detailed information、version or pictures.
If you haven't received a reply for a long time, you can join our slack and send your question to channel #troubleshooting

jieguangzhou commented 2 years ago

For a normal scenario, we can use the command like kubectl -f train.yaml or java sdk to apply a job to kubeflow
For the kubeflow pipeline, we can use a python script like the python task plugin.

We need to configure the k8s cluster information and synchronize the pods status

apache / dolphinscheduler