vLLM replaced their usages of the regular flash-attn library with their own vllm-flash-attn fork in vllm-project/vllm#4686, which, as of right now, is fairly easy to compile. This change compiles it from source for compatibility with the ml-containers/torch base images.
This is necessary to enable updating our vllm-tensorizer images to include the newest versions of vLLM.
Compile
vllm-flash-attnfrom SourcevLLM replaced their usages of the regular
flash-attnlibrary with their ownvllm-flash-attnfork in vllm-project/vllm#4686, which, as of right now, is fairly easy to compile. This change compiles it from source for compatibility with theml-containers/torchbase images.This is necessary to enable updating our
vllm-tensorizerimages to include the newest versions of vLLM.