Closed thkimYonsei closed 1 month ago
I have the same issue and solved it by just reloading the whole model on every iteration. Though now I am getting oom due to the memory not freeing after the second epoch, but I guess that is a different issue.
@Xirid, please try the following API to free engine memory https://deepspeed.readthedocs.io/en/latest/zero3.html#gpu-memory-management
我也遇到了在 for 循环中创建新的数据集和新的训练器,然后调用trainer.train()。 在 for 循环的第一次迭代中,训练工作正常。然而,在第二次迭代中,报错为错误: AttributeError: 'NoneType' object has no attribute 'numel' 尝试释放GPU,但没有用,释放GPU如下:# Free GPU memory consumed by model parameters ds_engine.empty_partition_cache()
Closing this issue due to lack of response. Please reopen if needed.
I was training LLaVA model using deepspeed zero3. What I want to do is continually train the model to different datasets. I create LLaVA model and in the for-loop, I create new dataset and new trainer, then calls
trainer.train()
. At the first iteration of the for-loop, the training works properly. However, at the second iteration, I got 1 warning and 1 error: warning:"Invalidate trace cache @ step XX: expected module XX, but got module XX"
error:AttributeError: 'NoneType' object has no attribute 'numel'
at the same location as this issueScreenshot:![image](https://github.com/microsoft/DeepSpeed/assets/55945983/2d1468af-43d3-4902-bc3b-ca270eca9964)
But when I simply change the deepspeed config to use zero2 instead of zero3, no error occurs. I want to use zero3 for larger batch size training. Can you help me out with this?
System info
zero3.json
that I used:zero2.json