Controlling stride of local attention window

Dao-AILab / flash-attention

Fast and memory-efficient exact attention

BSD 3-Clause "New" or "Revised" License

14.1k stars 1.31k forks source link

Open EomSooHwan opened 5 months ago

EomSooHwan commented 5 months ago

Hi, I am wondering if there is any way to change the stride of the local attention window.

For example, i-th query attends to keys in [i stride + seqlen_q - seqlen_k + win_size[0], i stride + seqlen_q - seqlen_k + win_size[1]]

tridao commented 5 months ago

No that's not supported.