Open littletomatodonkey opened 1 month ago
AFAIK custom attention masks are not supported in optimized implementations of attention (FlashInfer, FlashAttention), so this wouldn't be possible without also incurring a dramatic drop in performance.
AFAIK custom attention masks are not supported in optimized implementations of attention (FlashInfer, FlashAttention), so this wouldn't be possible without also incurring a dramatic drop in performance.
Thanks for your reply, i might have to realize the function myself.
🚀 The feature, motivation and pitch
I am workning on 4D attention mask input and LLM generateion process. Huggingface provides an interface for the 4D attention mask. Does vllm have any plan? https://github.com/huggingface/transformers/pull/27539
Alternatives
No response
Additional context
No response