Development Roadmap (2024 Q4) - Githubissues

sgl-project / sglang

SGLang is a fast serving framework for large language models and vision language models.

https://sglang.readthedocs.io/en/latest/

Apache License 2.0

5.21k stars 369 forks source link

Development Roadmap (2024 Q4) #1487

Open Ying1123 opened 11 hours ago

Ying1123 commented 11 hours ago

Here is the development roadmap for 2024 Q4. Contributions and feedback are welcome (Join Weekly Development Meeting). Previous 2024 Q3 roadmap can be found in #634.

Performance

[ ] Hide CPU overhead. See also a prototype.
[ ] Support speculative decoding
- Reference-based. See also a prototype.
- Draft model based.
- Medusa head #859
[ ] Efficient attention
- [ ] Sparse Attention #1459
- [ ] Parallel encoding
[ ] Multi-layer radix cache (GPU/CPU/Disk)
[ ] Faster grammar parsing library for constrained decoding

Parallelism

[ ] Finish sequence parallelism #1436
[ ] Support pipeline parallelism.
[ ] Implement a better load balancer for data parallelism. See also preble.
[ ] Support expert parallelism + data parallelism for DeepSeek models.
[ ] Support disaggregated serving to separate prefill and decoding.

Hardware Coverage

[ ] AMD optimizations. cc @HaiShaw
- FlashInfer support for AMD
- Setup CI (accuracy/performance) for AMD
[ ] Intel XPU support.
- 1480

Model Coverage

[ ] Multi-modal models
- QWen2-VL
- VILA
- Phi-vision
- FishSpeech audio model support
[ ] Language models
- Mamba models
[ ] Reward models

LoRA support

[ ] Performance optimizations for multi-LoRA serving #1471

Quantization

@zhyncs @ispobock

[ ] Torchao quantization
[ ] Turbomind operators integration
[ ] More CUTLASS mixed precision gemm integration
[ ] KV cache quantization (more formats + scaling factor)

Server API

[ ] Support directly taking embedding as inputs. #745
[ ] Add APIs for using the inference engine in a single script without launching a separate server. See also examples.
- 1127
[ ] Support endpoint other than OpenAI (Anthropic, Mistral) in the language frontend.
[ ] Better APIs to support RL trainers https://github.com/huggingface/trl

Observability

[ ] Integrate Grafana / Prometheus
- 1461

Others

[ ] Notebook-style interactive tutorials. @zhaochenyang20
[ ] Compiler mode optimizations for the language (e.g. support sending a full serialized SGL program to the server). @hnyls2002
[ ] Memory pool refactor to better support mixing different attention layers (e.g., interleaved window attention). @Ying1123
[ ] Linear layers refactor. Make vLLM an optional dependency. @zhyncs

fengyang95 commented 8 hours ago

Are there any plans to optimize long context latency?