fix(enable_qkv_fusion): minor fix for qkv fusion - Githubissues

InternLM / InternEvo

InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencies.

https://internevo.readthedocs.io/zh-cn/latest/?badge=latest

Apache License 2.0

311 stars 52 forks source link

fix(enable_qkv_fusion): minor fix for qkv fusion #340

Closed zigzagcai closed 2 months ago

zigzagcai commented 2 months ago

Motivation

(Fix for this PR, https://github.com/InternLM/InternEvo/pull/338) q_dim and kv_dim should be divided in the nn.Modules hooks, when tp/wp is enabled.
refine dispatch modules: move model.to(device) to inject_model_helper, to support larger parameter size

Modification

Please briefly describe what modification is made in this PR.

BC-breaking (Optional)

None

Use cases (Optional)

None

Checklist

Before PR:

[ ] Pre-commit or other linting tools are used to fix the potential lint issues.
[ ] Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests.
[ ] The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
[ ] The documentation has been modified accordingly, like docstring or example tutorials.

After PR:

[ ] If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects.
[ ] CLA has been signed and all committers have signed the CLA in this PR.