search

huggingface / text-generation-inference

Large Language Model Text Generation Inference
http://hf.co/docs/text-generation-inference
Apache License 2.0
8.63k stars 999 forks source link

DistServe support #2183

Closed icyxp closed 1 week ago

icyxp commented 1 month ago

Feature request

https://github.com/LLMServe/DistServe

Motivation

DistServe improves the performance of large language models (LLMs) serving by disaggregating the prefill and decoding computation. Existing LLM serving systems colocate the two phases and batch the computation of prefill and decoding across all users and requests. We find that this strategy not only leads to strong prefill-decoding interferences but also couples the resource allocation and parallelism plans for both phases. In DistServe, you can simply set the parallelism configs and scheduling strategies for the two phases and it will work just like a single instance which handles the KV-Cache communication and memory management automatically.

Your contribution

none

LysandreJik commented 1 month ago

Thanks for your request @farzanehnakhaee70! I'm making sure the team sees it.

Usually what drives an addition to our toolkit is the community excitement for it. Getting a lot of 👍 on your message will show us that it's important for a lot of people :)

github-actions[bot] commented 2 weeks ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.