search

kubeedge / ianvs

Distributed Synergy AI Benchmarking
https://ianvs.readthedocs.io
Apache License 2.0
115 stars 46 forks source link

Multimodal Large Model Joint Learning Algorithm: Reproduction Based on KubeEdge-Ianvs #123

Open CreativityH opened 4 months ago

CreativityH commented 4 months ago

What would you like to be added/modified: A benchmark suite for multimodal large language models deployed at the edge using KubeEdge-Ianvs:

  1. Modify and adapt the existing edge-cloud data collection interface to meet the requirements of multimodal data collection;
  2. Implement a Multimodal Large Language Model (MLLM) benchmark suite based on Ianvs;
  3. Reproduce mainstream multimodal joint learning (training and inference) algorithms and integrate them into Ianvs single-task learning;
  4. (Advanced) Test the effectiveness of multimodal joint learning in at least one of Ianvs' advanced paradigms (lifelong learning, incremental learning, federated learning, etc.).

Why is this needed: KubeEdge-Ianvs currently focuses on edge-cloud collaborative learning (training and inference) for a single modality of data. However, edge devices, such as those in autonomous vehicles, often capture multimodal data, including GPS, LIDAR, and Camera data. Single-modal learning can no longer meet the precise inference requirements of edge devices. Therefore, this project aims to integrate mainstream multimodal large model joint learning algorithms into KubeEdge-Ianvs edge-cloud collaborative learning, providing multimodal learning capabilities.

Recommended Skills: TensorFlow/Pytorch, LLMs, KubeEdge-Ianvs

Useful links: KubeEdge-Ianvs KubeEdge-Ianvs Benchmark Test Cases Building Edge-Cloud Synergy Simulation Environment with KubeEdge-Ianvs Artificial Intelligence - Pretrained Models Part 2: Evaluation Metrics and Methods Example LLMs Benchmark List awesome-multimodal-ml Awesome-Multimodal-Large-Language-Models

aryan0931 commented 2 months ago

Dear Aryan Yadav， My student Tianyu Tu will contact you for follow-up arrangements. He has a lot of experience in this research direction, and you can communicate more. We can discuss at any time. Bests, Chuang. Aryan Yadav @.> 于2024年9月6日周五 00:57写道： … Thanks, @CreativityH https://github.com/CreativityH, for selecting me as an LFX mentee for this term! I'm really excited to tackle this task and contribute to the project. Just wanted to check where I should reach out to you to discuss the next steps and further plans for working on this project idea. Looking forward to collaborating! 🙌 — Reply to this email directly, view it on GitHub <#123 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADURL3UKZMX2XJMOEBC2GCTZVCEPHAVCNFSM6AAAAABLF4MFJGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZSGIZDANZRGI . You are receiving this because you were mentioned.Message ID: @.>

Okay, sir, I received the mail from him.