search

alibaba / Pai-Megatron-Patch

The official repo of Pai-Megatron-Patch for LLM & VLM large scale training developed by Alibaba Cloud.
Apache License 2.0
674 stars 94 forks source link

llama3 报错 #210

Closed yangzhipeng1108 closed 5 months ago

yangzhipeng1108 commented 5 months ago

cd /workspace/Pai-Megatron-Patch/examples/llama3 sh run_pretrain_megatron_llama.sh \ dsw \ ../../ \ 8B \ 1 \ 8 \ 1e-5 \ 1e-6 \ 2048 \ 2048 \ 256 \ bf16 \ 2 \ 1 \ sel \ true \ true \ true \ true \ 100000 \ /workspace/data/oscar-en-10k-meg-llama_text_document \ /workspace/model_ckpts/Meta-Llama-3-8B-to-megatron-tp2-pp1 \ 100000000 \ 10000 \ /workspace/output_megatron_llama3 image

jerryli1981 commented 5 months ago

您好，收到，脚本内外网同步有问题，已修复：https://github.com/alibaba/Pai-Megatron-Patch/pull/211 然后刚又按照ReadMe重新测试下能跑通，辛苦再试试 https://github.com/alibaba/Pai-Megatron-Patch/blob/main/examples/llama3/README.md#Megatron-LM-Dense%E6%A8%A1%E5%9E%8B%E8%AE%AD%E7%BB%83%E6%B5%81%E7%A8%8B

yangzhipeng1108 commented 5 months ago

厉害