[Roadmap] vLLM Roadmap Q1 2024

[zhuohan123]

This document includes the features in vLLM's roadmap for Q1 2024. Please feel free to discuss and contribute to the specific features at related RFC/Issues/PRs and add anything else you'd like to talk about in this issue.

In the future, we will publish our roadmap quarterly and deprecate our old roadmap (#244).

• OSS General
  • [x] Better benchmark scripts and standards (#2433)
  • [ ] Improve documentation
  • CI/CD Testing and release process
  • [ ] Make model and kernel tests working on current CI
  • [ ] Automate release process
  • [ ] Dev experience
  • [ ] Explore Apple Silicon via Torch or MLX or llama cpp
  • [ ] Cached and parallel build system (#2654)
• Frontend
  • [x] Support structured output (contact: @simon-mo)
  • [ ] Optimize the performance of the API server
• Scheduling
  • [ ] Chunked prefill / dynamic splitfuse (#1562)
  • [ ] Speculative decoding (#2188, #2607, merging plan, contact: @LiuXiaoxuanPKU)
  • [x] Automatic prefix caching (#2614, contact: @zhuohan123)
  • [ ] Disaggregated prefill / splitwise (#2472)
• Kernel performance optimization
  • [ ] Quantization kernel optimization
  • [ ] Support FP8 (#2461)
  • [ ] MoE kernel optimization (#2453, #2542, and more)
  • [ ] H100 performance (#2107)
  • [ ] AMD MI300x Performance
  • [ ] MQA kernel (#1880)
  • [ ] Port FlashInfer to vLLM (#2767)
  • [ ] Kernel for sampler
  • Hardware support vLLM team is working with the following hardware vendors:
  • [x] AWS Inferentia (#1866, #2569)
  • [ ] Google TPU
  • [ ] Intel Gaudi
  • [ ] Intel GPU/CPU (#2378)
  • Model support
  • [x] Multi-modal models (#775, #1265, #2153, #2563)
  • [ ] Encoder-decoder models (#187, #404)
  • [ ] Embedding models (#458, #742)
  • Future-proofing vLLM
  • [ ] torch.compile support
  • [ ] Implement extensible scheduler and memory manager

[sandangel]

Is it possible to support mlx for running inference on Mac devices? That would simplify the local development and running on cloud.

[AguirreNicolas]

As mentioned in #2643, it would be awesome to have vLLM /completions & /chat/completions endpoints both supporting logprobs to run lm-eval-harness.

[PeterXiaTian]

please take attention with "Evaluation of Accelerated and Non-Accelerated Large Model Output",it is very important and make sure they are always same

[jrruethe]

As mentioned in #2643, it would be awesome to have vLLM /completions & /chat/completions endpoints both supporting logprobs to run lm-eval-harness.

Agree 100%, the ability to use lm-eval-harness is very much needed

[casper-hansen]

2767 I suggest adding this to the roadmap as it's one of the more straight forward optimizations (someone already did the optimization work).

[jalotra]

https://github.com/vllm-project/vllm/issues/2573 this talks about optimising api server Optimize the performance of the API server

[cyc00518]

Please support ARM aarch-64 architecture.

[Tint0ri]

https://github.com/vllm-project/vllm/issues/1253

please consider support streamingllm

[kanseaveg]

Any update for PEFT?

please consider support huggingface peft, thank you. #https://github.com/vllm-project/vllm/issues/1129

[ekazakos]

Would you consider adding support for earlier ROCm versions, e.g. 5.6.1.? Thank you!

[pabl-o-ce]

If is possible exl2 support thank you <3

[hmellor]

97 should be added to the automating the release process section

[jrruethe]

Also, the ability to use Guidance/Outlines via logit_bias! And +1 to EXL2 support

[busishengui]

support W8A8

[simon-mo]

Let's migrate out discussion to #3861