Hi, thanks for your great works.
When I used ZeRO3 configuration, the training results was ok.
However, when switching to the ZeRO2 configuration, the loss equals zero.
The only difference in the training arguments is the ZeRO stage configuration, and I trained the models on 8x A800 GPUs.
Are there any potential reasons for this discrepancy?
Hi, thanks for your great works. When I used ZeRO3 configuration, the training results was ok. However, when switching to the ZeRO2 configuration, the loss equals zero. The only difference in the training arguments is the ZeRO stage configuration, and I trained the models on 8x A800 GPUs. Are there any potential reasons for this discrepancy?