vLLM replaced their usages of the regular flash-attn library with their own vllm-flash-attn fork in vllm-project/vllm#4686, which, as of right now, is fairly easy to compile. This change compiles it from source for compatibility with the ml-containers/torch base images.
This is necessary to enable updating our vllm-tensorizer images to include the newest versions of vLLM.
Compile
vllm-flash-attn
from SourcevLLM replaced their usages of the regular
flash-attn
library with their ownvllm-flash-attn
fork in vllm-project/vllm#4686, which, as of right now, is fairly easy to compile. This change compiles it from source for compatibility with theml-containers/torch
base images.This is necessary to enable updating our
vllm-tensorizer
images to include the newest versions of vLLM.