feat(examples): Add a simple FlashAttention Implementation.

TiledTensor / TiledCUDA

TiledCUDA is a highly efficient kernel template library designed to elevate CUDA C’s level of abstraction for processing tiles.

MIT License

158 stars 10 forks source link

feat(examples): Add a simple FlashAttention Implementation. #123

Closed KuangjuX closed 3 months ago

KuangjuX commented 3 months ago

Draft. Don't review

KuangjuX commented 3 months ago

I want to merge this part of the code first. Although it hasn't undergone rigorous correctness verification, it's already quite substantial. I will add test code in the next PR.