Zero3 train: Invalidate trace cache @ step 391: expected module 25, but got module 5

X-PLUG / mPLUG-Owl

mPLUG-Owl: The Powerful Multi-modal Large Language Model Family

https://www.modelscope.cn/studios/damo/mPLUG-Owl

MIT License

2.25k stars 171 forks source link

Zero3 train: Invalidate trace cache @ step 391: expected module 25, but got module 5 #208

Open Zhiyuan-Fan opened 7 months ago

Zhiyuan-Fan commented 7 months ago

{'loss': 0.8527, 'learning_rate': 1.282051282051282e-07, 'epoch': 0.0}
{'loss': 0.9276, 'learning_rate': 2.564102564102564e-07, 'epoch': 0.0}
0%| | 2/5194 [00:26<18:22:29, 12.74s/it] Invalidate trace cache @ step 391: expected module 25, but got module 5