Open NouamaneTazi opened 1 year ago
Aah here we go. Is flash attention merged into the original repo? I saw Tri Dao had opened a PR
It's not merged yet to Megatron-LM. But I hope it will soon be :)
@NouamaneTazi Hi. Have you tested the correctness of this implementation of Flash Attention?
This PR aims to add an option to use FlashAttention. Inspired by https://github.com/NVIDIA/Megatron-LM/pull/267
cc @thomasw21