Refactor communicate groups and Block - Githubissues

OpenBMB / BMTrain

Efficient Training (including pre-training and fine-tuning) for Big Models

Apache License 2.0

560 stars 77 forks source link

Refactor communicate groups and Block #159

Closed zkh2016 closed 1 year ago

zkh2016 commented 1 year ago

Pull Request Template

Issue Reference

Please mention the issue number if applicable, or write "N/A" if it's a new feature. N/A

Description

Please describe your changes in detail. If it resolves an issue, please state how it resolves it.

重构通信组，支持各种策略混合使用
重构Block
当前不支持初始化开启pipe_size，但是模型中没用使用PipelineTransformerBlockList情况
使用PipelineTransformerBlockList如果传入的是Block，必须指定mode是PIPE
不支持TransformerBlockList中插入多个重复的block

Type of Change

[ ] Bug fix (non-breaking change which fixes an issue)
[x] New feature (non-breaking change which adds functionality)
[ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
[ ] This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce.

Checklist

[x] I have read the CONTRIBUTING document.
[x] My code follows the code style of this project.
[x] My change requires a change to the documentation.
[x] I have updated the documentation accordingly.
[x] I have added tests to cover my changes.
[x] All new and existing tests passed.

Additional Information

Any additional information, configuration, or data that might be necessary for the review.