search

lucidrains / perceiver-pytorch

Implementation of Perceiver, General Perception with Iterative Attention, in Pytorch
MIT License
1.1k stars 134 forks source link

FlashAttention implementation #65

Open kklemon opened 1 year ago

kklemon commented 1 year ago

Inspired by this work, I implemented the Perceiver architecture with out-of-the-box FlashAttention support. It offers a great speedup over a naive implementation and up to 16x increased input sequence lengths for the same hardware.

You can find the project under fast-perceiver.

lucidrains commented 1 year ago

@kklemon very nice! 🚀

lucidrains commented 1 year ago

80vdww