search

intel / xFasterTransformer

Apache License 2.0
374 stars 65 forks source link

support vllm? #250

Closed leiwen83 closed 5 months ago

leiwen83 commented 8 months ago

Hi,

Do you consider support vllm, which is actually a llm serving standard nowadays?

Thx~

Duyi-Wang commented 7 months ago

I writed a draft to support vllm. https://github.com/Duyi-Wang/vllm But not support distributed and continues batching yet.

Duyi-Wang commented 5 months ago

A fork of vLLM has been created to integrate the xFasterTransformer backend, maintaining compatibility with most of the official vLLM's features. https://github.com/intel/xFasterTransformer/blob/main/serving/vllm-xft.md