search

HuangLK / transpeeder

train llama on a single A100 80G node using 🤗 transformers and 🚀 Deepspeed Pipeline Parallelism
Apache License 2.0
208 stars 18 forks source link

support bf16？ #23

Open lw3259111 opened 1 year ago

lw3259111 commented 1 year ago

how to support bf16?

HuangLK commented 1 year ago

ds may have some issues about bf16, see this. Anyway, you can try.

HuangLK commented 1 year ago

A recent pr about pp+zero1+bf16

lw3259111 commented 1 year ago

A recent pr about pp+zero1+bf16

tks, How to run on multiple nodes？

zhongwenjie01 commented 1 year ago

when i use bf16+zero1+pp,offload doesn't work