Open sangstar opened 2 months ago
vllm-tensorizer hasn't had updates since vLLM's formal adoption of tensorizer model loading. An update to build for the most recent commit to vLLM that includes sharded tensorizer support is presented, along with some fixes to successfully build vLLM with recent updates to the source code. These include:
vllm-tensorizer
tensorizer
cmake
xformers
0.0.26.post1
flash-attn
vllm-tensorizer
hasn't had updates since vLLM's formal adoption oftensorizer
model loading. An update to build for the most recent commit to vLLM that includes sharded tensorizer support is presented, along with some fixes to successfully build vLLM with recent updates to the source code. These include:tensorizer
)cmake
xformers
version to0.0.26.post1
flash-attn
, which is built here from source