search

huggingface / nanotron

Minimalistic large language model 3D-parallelism training
Apache License 2.0
1.14k stars 107 forks source link

Ring attention #181

Open zzhhjjj opened 4 months ago

zzhhjjj commented 4 months ago

Ring attention for training on long sequences. Similar to Megatron context parallel. Idea from https://github.com/zhuzilin/ring-flash-attention