About the vq loss when training the codebook

[wangzheng1209]

1. Can you explain why "it is normal for the loss to increase before decreasing"?
2. And I want to ask if it has some thing to do with the learning rate "2.0" set in the l2_32_smoothSS.json.

[evonneng]

The loss increasing then decreasing while training the codebook was just something we saw occur across all instances of training for the VQ codebook. It is unclear why exactly it happens. We did do sweeps across the learning rate parameter, and noted that for smaller LR, the model was never able to make it past this peak and saturated at some suboptimal minima. On the other hand, larger LR resulted in much more unstable training that never saturated. Hope this bit helps!