jorgeantonio21
commented
3 weeks ago In order to further guide the work on this issue, we suggest the following:
- [ ] Get familiar with the main ideas in the paper.
- [ ] Understand how the Llama architecture works, at a high level (how KV cache is used, how attention is calculated, etc).
- [ ] Get familiar with the cuda code in attention_kernels.cu.
- [ ] Once you are familiar with the code in attention_kernels.cu, you can try your own implementation, or simply add the file here. Make sure, you include all necessary dependencies.
- [ ] Create a new PR, tagging this issue, and explaining briefly your understanding of paged attention and how it can be used to optimized Llama serving AI inference.
RajeshRk18
g4titanx
fishonamos
Josh-121
No-bodyq
Following the paged attention paper, add cuda kernels for the Llama model. Cuda kernels for the Llama architecture have been widely implemented in the open source community.
Moreover, paged attention kernels can be found in the original vLLM implementation here. Of special interest for us is the cuda implementation in attention_kernels.cu, which implements the attention logic using the pagination algorithms (described in the reference paper above).
NOTE: Issue 1 must be completed in order to progress onto issues 2,3.4