modify clip grad - Githubissues

Oneflow-Inc / oneflow

OneFlow is a deep learning framework designed to be user-friendly, scalable and efficient.

http://www.oneflow.org

Apache License 2.0

5.79k stars 658 forks source link

modify clip grad #10441

Closed hanwen-sun closed 2 months ago

hanwen-sun commented 3 months ago

修改clip_grad中tensor parallel的通信策略:

L2 norm计算修改:
- 每张卡先在本地计算梯度的squaresum
- 调用all_reduce_sum, 每张卡得到累加后的squaresum
- 开根号
TODO:
- 更好的写法;

github-actions[bot] commented 3 months ago

Code got formatted by CI. Please request CI again if you still want to have this PR merged. If the PR is from a forked repo, please download the patch files from the GitHub Actions web page and apply them locally.

levi131 commented 2 months ago

问题在 https://github.com/Oneflow-Inc/oneflow/pull/10443 中解决