Add Splitwise: prompt and token phase separation

[goiri]

We have built the system described in http://aka.ms/splitwise Splitwise splits the prompt and token phases to run in different servers. This leverages the differences between these two phases to improve throughput. We have an internal prototype on top of an internal vLLM branch. This issue tracks the effort to open source this prototype and make it part of the official vLLM.

This includes:

• Add MSCCL++ support https://github.com/microsoft/mscclpp
• Add per-layer KV-cache transfer
• Coordination across prompt and token servers
• Documentation

[goiri]

This was asked in https://github.com/vllm-project/vllm/issues/2370.

[irasin]

LGTM, I was wondering when can we use it in vllm?

[goiri]

@irasin, @aashaka is doing some cleanup and refactoring and will be posting the PRs in the next few weeks. We will be updating this issue (and linking the PRs) with the progress.

[adney11]

Hi All,

Just wanted to check in and see if there is any update on Splitwise's implementation in vLLM, and if this internal prototype codebase can be released?

Thank you!

[aashaka]

This has now been released in PR https://github.com/vllm-project/vllm/pull/2809. @adney11, @irasin