search

kubeedge / kubeedge

Kubernetes Native Edge Computing Framework (project under CNCF)
https://kubeedge.io
Apache License 2.0
6.8k stars 1.73k forks source link

Integrate KubeEdge, Sedna, and Volcano for High-Performance Training Task Scheduling #5762

Open Shelley-BaoYue opened 3 months ago

Shelley-BaoYue commented 3 months ago

What would you like to be added/modified:

Combines KubeEdge with its subproject Sedna, and integrates Volcano's scheduling capabilities to deploy an training task at edge. The final outcomes of this project would be:

Why is this needed:

KubeEdge and Sedna have already enabled edge-cloud collaborative training and collaborative inference capabilities. We aim to explore and foster collaborations with more communities to provide enhanced AI capabilities. By integrating Volcano, we aim to achieve high-performance scheduling within the cloud-edge collaborative framework, thereby pushing the boundaries of what can be achieved in distributed AI and edge computing.

SargamPuram commented 3 months ago

Hi @Shelley-BaoYue ,

I'm excited about the opportunity to contribute to the "Integrate KubeEdge, Sedna, and Volcano for Efficient Task Scheduling" project. My name is Sargam, and I have experience in edge computing, cloud-native technologies, and AI. Here’s my proposed approach:

Proposed Approach

  1. Deploy Training Task Using KubeEdge and Sedna:

I will set up and successfully deploy a training task using KubeEdge and Sedna. This will include creating and documenting an example in the KubeEdge examples repository, demonstrating the edge-cloud collaborative training and inference capabilities.

  1. Integrate Volcano for High-Performance Scheduling:

I will integrate Volcano within Sedna's architecture to enable high-performance scheduling of training tasks. This will involve configuring Volcano's scheduling capabilities to optimize resource utilization and reduce latency within the edge-cloud collaborative framework.

  1. Deploy Kubeflow within KubeEdge:

I will deploy Kubeflow within the KubeEdge architecture and complete the deployment of a training task. This process will be documented in a detailed blog post, providing insights and guidance on the integration process and showcasing the enhanced AI capabilities achieved through this setup. Looking forward to your feedback and the way forward to contributing! Also curious to know are there any pretests for this? Or should I directly apply on the LFX Mentorship portal.

Best, Sargam

peeplika commented 3 months ago

Hello, @Shelley-BaoYue I am interested in this project, I am currently looking into these repositories and getting myself familiarized. If you could provide any relevant resources that I should be going through or if there is any communication channel for discussion, that would be really helpful. Thank You

Electronic-Waste commented 3 months ago

Hi @Shelley-BaoYue ,

I'm Shao Wang, a freshman graduate student in Shanghai Jiao Tong University, majoring in Computer Science. I'm interested in applying for KubeEdge project "Integrate KubeEdge, Sedna, and Volcano for Efficient Task Scheduling" in LFX Mentorship.

I think, What makes me competent for the application is that I have some professional experience and projects concerned with Go, Kubernetes, and Kubeflow. This guarantees I can understand and contribute to KubeEdge with less effort learning its pre-knowledge. My professional experiences are listed in the following lines:

  1. Minik8s: A naive Kubernetes-like container orchestration system.
  2. Satellite-SDN: A large-scale heterogenous network emulation system based on Kubernetes.
  3. Internship in BondiTech as an infrastructure engineer, building a new quotes storage system from scratch based on Kubernetes and ClickHouse, which enhances the efficiency of data analysis business by 10x times.

What's more, I'm also an active contributor to Kubeflow's subproject Katib and Training-Operator. Thus, I think I can successfully accomplish your optional task "Successfully deploy Kubeflow within the KubeEdge architecture and complete the deployment of a training task, with a blog post documenting the process". And I hold a firm belief that this step will surely make the KubeEdge community bond with our Kubeflow community and boost our collaboration with each other.

I'm currently investigating the KubeEdge code base. If you could offer me some relevant resources, it would help me understand KubeEdge better.

Shelley-BaoYue commented 2 months ago

Hi, @Electronic-Waste
Congratulations on becoming a mentee of LFX KubeEdge! 😄 Please go ahead on this project. If you have any questions, feel free to communicate in the issues section, or join our community meetings every Wednesday at 4 PM. https://zoom.us/j/4167237304#success

Electronic-Waste commented 2 months ago

Thanks @Shelley-BaoYue It's a pleasure working with the KubeEdge community this fall! I'll investigate this project. But before that, may I ask whether I need to write the design proposal like KEP and push it to doc dir or not? Could you tell me which way you prefer?

Electronic-Waste commented 2 months ago

/assign

Shelley-BaoYue commented 2 months ago

Integrate Volcano within Sedna's architecture to achieve high-performance scheduling of training tasks may need a proposal. You can share your idea at the community meeting before you submit the proposal.

Electronic-Waste commented 2 months ago

@Shelley-BaoYue I'll first draft my proposal. Thank you for your clarification!

Electronic-Waste commented 1 month ago

I've created a Google Doc for this project. I will document the design proposal and my progress in it :)

https://docs.google.com/document/d/1qS9CvupcGOtdHIZDZT6O87t1GQ8O3F-AxYS4i2nwPpE/edit?usp=sharing