training failed when change the VectorQuantizer to SimVQ

[lucidrains]

seems like you figured it out, your had accept image hparam turned on

[JohnHerry]

seems like you figured it out, your had accept image hparam turned on

Yes， when I turn off the image param, the training can run. but it seems that we should tune the training params like learning-rate and loss's weights. because the training lost into gradiant NaN after only 2-3 thousands steps. our primary loss weights to codebook-loss, adv-loss, reconstruction-loss is 1, 1, 10. I have changed the loss weights to 1000, 1, 10, I do not know which is the good config, just trying. need more suggestion about it.

[JohnHerry]

seems like you figured it out, your had accept image hparam turned on

Yes， when I turn off the image param, the training can run. but it seems that we should tune the training params like learning-rate and loss's weights. because the training lost into gradiant NaN after only 2-3 thousands steps. our primary loss weights to codebook-loss, adv-loss, reconstruction-loss is 1, 1, 10. I have changed the loss weights to 1000, 1, 10, I do not know which is the good config, just trying. need more suggestion about it.

It seems that 1000, 1, 10 is not a good config. When rec_loss go down, the codebook_loss become very large, and when the codebook_loss go down, the rec_loss become very large. Is there any suggestion for improve?