fix scale loss logic - Githubissues

OpenBMB / BMTrain

Efficient Training (including pre-training and fine-tuning) for Big Models

Apache License 2.0

548 stars 74 forks source link

fix scale loss logic #194

Closed MayDomine closed 5 months ago

MayDomine commented 5 months ago

Fix scale loss logic

Description

Scale formula changes from loss = loss / dp_size to loss / (dp_size * tp_size).

Type of Change

[x] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce.

Checklist

[x] I have read the CONTRIBUTING document.
[x] My code follows the code style of this project.
[] My change requires a change to the documentation.
[ ] I have updated the documentation accordingly.
[ ] I have added tests to cover my changes.
[ ] All new and existing tests passed.