search

lucidrains / ring-attention-pytorch

Implementation of 💍 Ring Attention, from Liu et al. at Berkeley AI, in Pytorch
MIT License
474 stars 27 forks source link

striped causal version of `ring_flash_attn_cuda` is not working #11

Closed lucidrains closed 7 months ago

lucidrains commented 7 months ago

^

lucidrains commented 7 months ago

it works, but only with 1 bucket per machine for now

lucidrains commented 7 months ago

it completely works now