search

feifeibear / long-context-attention

USP: Unified (a.k.a. Hybrid, 2D) Sequence Parallel Attention for Long Context Transformers Model Training and Inference
Apache License 2.0
313 stars 19 forks source link

convert input tensor layout inside stripe and zigzag ring attention #46

Closed feifeibear closed 5 months ago

feifeibear commented 5 months ago

Now, we have to prepare a correct input layout for zigzag and stripe before execute forward().