Evaluate vLLM

[lmitlaender]

This issues goal is to evaluate the vllm-project for llm serving (in comparison to ollama).

vllm source code: https://github.com/vllm-project/vllm

[lmitlaender]

1. Difference: vLLM only works on Linux and will lead to an error on windows on installation. At the same time vLLM only supports GPU Inference which shouldnt be a too big limitation.

[lmitlaender]

2. Provides faster SOTA Optimization support like TensorParallelism, Continuous Batching
3. Works with most standard Huggingface Model Architectures, and supports AWQ Quantization
4. Integrates with Ray Serve for Full Scale distributed computing

[lmitlaender]

5. Isn't supposed to do any prompt magic, stronger seperation of concerns

[lmitlaender]

6. Supports OpenAI API serving

https://www.youtube.com/watch?v=TJ5K1CO9Wbs See parts of this video for some of the speedups that vllm can support

[lmitlaender]

Might be interesting: https://github.com/ray-project/ray-llm integrates vllm with ray in a single solution combining multiple optimizations