Issue towards prompt increase of accuracy in your training log file 'vssm_base_0229.txt'

[YitianShi]

Hi,

recently I was checking your log file to compare with my model, I found that in your vssm_base_0229.txt the accuracy was dropping towards 35%, while you reload the model in epoch 60 and the accuracy increased to 78.665% promptly even if the highest accuracy was around 62%. Could explain what you did? Thank you very much!

Before reloading:

After reloading:

[MzeroMiko]

There's nothing we can do in just 30s 😂 (00:10:59 -> 00:11:29)

The thing is, I tried using fp16 as the output data format of SS2D in evaluating to save a little time, but I failed when the model is still on training. (Maybe during training, far from converging, the model is too sensitive, while after training converges, the model becomes not that sensitive and can be used in total fp16 format.)

Then I found that the log is not human-friendly, so I killed it and resumed it.