feat(dataloader): Implement megatron dataloader and mocked dataloader - Githubissues

InternLM / InternEvo

InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencies.

https://internevo.readthedocs.io/zh-cn/latest/?badge=latest

Apache License 2.0

285 stars 47 forks source link

feat(dataloader): Implement megatron dataloader and mocked dataloader #323

Closed zigzagcai closed 2 weeks ago

zigzagcai commented 2 weeks ago

This PR's main functionality is okay and runnable, but still need some refinement.

Motivation

Support megatron dataloader type: When users want to use InternEvo framework to train over megatron tokenized datasets.
Support mocked dataloader type: When users want to conduct precision alignment experiment to ensure that the loaded data is completely consistent.

Modification

internlm/data/megatron/* internlm/data/mocked/*

BC-breaking (Optional)

None

Use cases (Optional)

None

Checklist

Before PR:

[ ] Pre-commit or other linting tools are used to fix the potential lint issues.
[ ] Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests.
[ ] The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
[ ] The documentation has been modified accordingly, like docstring or example tutorials.

After PR:

[ ] If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects.
[ ] CLA has been signed and all committers have signed the CLA in this PR.

sunpengsdu commented 2 weeks ago

LGTM