Support sliding window attention in FA3

Dao-AILab / flash-attention

Fast and memory-efficient exact attention

BSD 3-Clause "New" or "Revised" License

13.86k stars 1.28k forks source link

Open lin-ht opened 1 month ago

lin-ht commented 1 month ago

Thank you for the great work on FA3. I am wondering if FA3 will support sliding window attention soon as FA2 does?

tridao commented 1 month ago

There's a PR on that

lin-ht commented 1 month ago

Wow~ Thanks! I put the PR link here for future reference: https://github.com/Dao-AILab/flash-attention/pull/1197

ipiszy commented 1 month ago

Sorry I deleted the original branch by accident. This is the new PR: https://github.com/Dao-AILab/flash-attention/pull/1233.