search

dome272 / MaskGIT-pytorch

Pytorch implementation of MaskGIT: Masked Generative Image Transformer (https://arxiv.org/pdf/2202.04200.pdf)
MIT License
398 stars 34 forks source link

Learning rate & its scheduling #2

Closed LeeDoYup closed 2 years ago

LeeDoYup commented 2 years ago

I cannot find the specific value of learning rate and how the author schedule to change the learning rate over epochs. How do you implement and reproduce the results in the paper?

dome272 commented 2 years ago

To be honest, I was also a bit disappointed that there were no implementation details given by the authors. So for, I did not do any hyperparameter tuning and just set the learning rate to be the same as used in the original VQGAN paper for the transformer part. That also means that I didn't experiment with any lr-schedules so feel free to experiment with it I would say. And for the last question, I did not reproduce the results yet and they are still sitting on the TODO list. For now all I did was implementing the Bidirectional Transformer into the VQGAN code. Furthermore, the authors refered to two papers when talking about the Bidirectional Transformer which might lead to some serious performance improvement. I simply took my own standard implementation of BERT. So again, feel free to do some optimization there.

LeeDoYup commented 2 years ago

Thanks for the reply. I agree with you that the paper does not include the implementation details.