search

HuangLK / transpeeder

train llama on a single A100 80G node using 🤗 transformers and 🚀 Deepspeed Pipeline Parallelism
Apache License 2.0
208 stars 18 forks source link

Add ntk, flash-attn2 and support llama2 #39

Closed JY-Ren closed 1 year ago

JY-Ren commented 1 year ago
  1. add ntk and flash-attn2
  2. support llama2
  3. refine rope
  4. update requriremnet.txt and convert2hf script