-
Nice work! Really want to see it supports megatron.
-
follow:https://github.com/alibaba/Pai-Megatron-Patch/blob/main/examples/qwen1_5/README.md#megatron-lm-dense%E6%A8%A1%E5%9E%8B%E6%A0%BC%E5%BC%8F%E8%BD%AC%E6%8D%A2
报错如下:看起来是pytorch_model.bin和safetens…
-
在应用完补丁执行pretrain_gpt.py遇到的问题
Traceback (most recent call last):
File "pretrain_gpt.py", line 126, in
pretrain(train_valid_test_datasets_provider, model_provider, forward_step,
File "/work…
-
cd /workspace/Pai-Megatron-Patch/examples/llava/
sh run_pretrain_megatron_llava.sh \
dsw \
/workspace/Pai-Megatron-Patch \
7B \
4 \
32 \
1e-3 \
1e-4 \
2048 \
2048 \
0 \
bf16 \
…
-
Hi guys,
I followed [this guide](https://huggingface.co/docs/accelerate/en/usage_guides/megatron_lm) to pre-train a GPT-2 model using Accelerate with Megatron as backend. The current version of Meg…
-
### System Info
transformers 4.40.0
python 3.10
### Who can help?
@ArthurZucker
@Narsil
@SunMarc
### Information
- [ ] The official example scripts
- [x] My own modified scripts
…
-
https://github.com/NVIDIA/Megatron-LM/blob/54f1f78529cbc2b9cddad313e7f9d96ac0420a27/megatron/legacy/model/multiple_choice.py#L42
-
File "/gpfs01/unifiedcsi/gpfs/csi-dfs-ti-platform-fs/wcp/vllm_test/torchtune/Pai-Megatron-Patch/examples/llama2/pretrain_megatron_llama.py", line 110, in train_valid_test_datasets_provider
[rank2]: …
-
使用的是 README.md 中推荐的镜像
目前发现有两个问题:
**问题一:**
> megatron_core 0.7.0
> Pai-Megatron-Patch 0.8.3 / Pai-Megatron-Patch 0.9.0 都试过
按照 examples/qwen2 下面的README.md,对 Qwen2-1.5B 进行的操作(A100 4ka):
```
1…
-
> Using ndtimeline-tool to Monitor Megatron-GPT I want to use the ndtimeline-tool to monitor the computation and communication of each rank in Megatron-GPT. I have two concerns:
>
> 1…