[Feature] Multi-instance deployment

sgl-project / sglang

SGLang is a fast serving framework for large language models and vision language models.

https://sglang.readthedocs.io/en/latest/

Apache License 2.0

5.62k stars 435 forks source link

Open vkc1vk opened 4 days ago

vkc1vk commented 4 days ago

[X] 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
[X] 2. Please use English, otherwise it will be closed.

I want to host multiple instances of Llama 3 8B across multiple A100 GPUs (with load balancing handled). Is there a way to do this currently?

n/a

merrymercy commented 3 days ago

vkc1vk commented 3 days ago

Thanks! Is this option available with sgl.Engine as well (Batched offline mode)

ByronHsu commented 3 days ago

Yes you can use sgl.Engine with the current simple DP. We are working on the full-fledged version.