FlashAttention Pytorch Integration

Dao-AILab / flash-attention

Fast and memory-efficient exact attention

BSD 3-Clause "New" or "Revised" License

13.57k stars 1.24k forks source link

FlashAttention Pytorch Integration #1005

Open DianCh opened 3 months ago

DianCh commented 3 months ago

Hi authors! Hi, I’m trying to experiment and make tweaks and potential upgrades to FlashAttention, and wondering if this repoi or the Pytorch source code is the best place to start. Does the Pytorch integration copy-paste/pull from this original FlashAttention repo, or there are implementation changes made along with the integration - any chance you know about it?

Thanks!

tridao commented 3 months ago

The kernels are copy-pasted afaik.