search

hpcaitech / ColossalAI

Making large AI models cheaper, faster and more accessible
https://www.colossalai.org
Apache License 2.0
38.79k stars 4.34k forks source link

[FEATURE]: serving multiple models #3491

Open dlzou opened 1 year ago

dlzou commented 1 year ago

Describe the feature

Currently, does Colossal-AI have support or ongoing work for deploying multiple models concurrently, possibly using the Ray framework?

For context, I’m doing a course/research project related to multi-model serving, inspired by the AlpaServe paper. My professor referred me to Colossal-AI, and I would be interested in incorporating it in my project.

ucas010 commented 1 year ago

this will make it great

JThh commented 1 year ago

Interesting feature. Would you like to attempt to make an implementation roadmap?

dlzou commented 1 year ago

As I'm still learning how to use Colossal-AI, perhaps someone with more experience can lay out a general roadmap for this.

My project is more limited in scope, and the usage I'm interested in likely deviates from the broader use case. In particular, I'm looking to colocate multiple models on a set of devices like figure 1 in the AlpaServe paper shows.

binmakeswell commented 1 year ago

Hi @dlzou Yes, we are considering it, https://github.com/orgs/hpcaitech/projects/17/views/1