Does the training support standalone multi-card, distributed and larger models like qwen2.5 72b?

[wphtrying]

System Info

训练是否支持分布式以及更大模型比较qwen72b？

Who can help?

@morning9393

Information

• [X] The official example scripts
• [X] My own modified scripts

Tasks

• [x] An officially supported task in the codebase (such as scrips/, ...)
• [ ] My own task or dataset (give details below)

Reproduction

训练是否支持分布式以及更大模型比较qwen72b？

Expected behavior

期待训练成功

[linyaoyang]

同问，train_llm.sh训练1.5b的模型需要多少显存？能否用多卡训练？如何训练参数规模比较大的模型？

[wphtrying]

同问，train_llm.sh训练1.5b的模型需要多少显存？能否用多卡训练？如何训练参数规模比较大的模型？

看了下PPO的代码，应该是不支持单机多卡和分布式

[git-yutong]

哭了，空有两三台4090就是推理不进去。