[Bug] 大数据量训练出现内存不足 - Githubissues

OpenGVLab / InternVL

[CVPR 2024 Oral] InternVL Family: A Pioneering Open-Source Alternative to GPT-4o. 接近GPT-4o表现的开源多模态对话模型

https://internvl.readthedocs.io/en/latest/

MIT License

6.05k stars 471 forks source link

[Bug] 大数据量训练出现内存不足 #618

Open zxc351200 opened 1 month ago

zxc351200 commented 1 month ago

Checklist

[X] 1. I have searched related issues but cannot get the expected help.
[X] 2. The bug has not been fixed in the latest version.
[X] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

Describe the bug

使用2000w数据训练是正常的，但是使用5000w数据训练就会出现内存不足。 dataloader_num_workers设置为1也会报错。目前单机内存是1600G。请问有什么比较好的解决方法吗，internvl应该训练过更多数据吧？

Reproduction

sh 8b_full.sh

Environment

torch 2.4

Error traceback

No response

zxc351200 commented 1 month ago

dataloader_num_workers设置为0勉强可以正常训练，还有其他解决方法吗？

Weiyun1025 commented 1 month ago

我们会对数据在不同rank上做切分，每个rank只读取一部分数据，具体可以参考这里的代码