[Feature Request]: support for vAttention style paging for attention

NVIDIA / TensorRT-LLM

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.

https://nvidia.github.io/TensorRT-LLM

Apache License 2.0

8.13k stars 897 forks source link

[Feature Request]: support for vAttention style paging for attention #1944

Open thecheekygeek opened 1 month ago

thecheekygeek commented 1 month ago

System Info

Who can help?

@ncomly-nvidia https://arxiv.org/abs/2405.04437

This is currently a PR on vLLM as well. Can we also get it on here too, to support dynamic memory management for the non paged kernel? Thanks

Information

[ ] The official example scripts
[ ] My own modified scripts

Tasks

[ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[ ] My own task or dataset (give details below)

Reproduction

Expected behavior

actual behavior

additional notes

thecheekygeek commented 1 month ago

@ncomly-nvidia @AdamzNV @byshiue looks like there is also an official release: https://github.com/microsoft/vattention

github-actions[bot] commented 2 weeks ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 15 days."