Question about the mask token id and sos token id

dome272 / MaskGIT-pytorch

Pytorch implementation of MaskGIT: Masked Generative Image Transformer (https://arxiv.org/pdf/2202.04200.pdf)

MIT License

398 stars 34 forks source link

Question about the mask token id and sos token id #9

Closed larryzhang23 closed 2 years ago

larryzhang23 commented 2 years ago

Hi, In transformer.py, I find mask_token_id is set to be args.num_image_tokens. Shouldn't it be the args.num_codebook_vectors? I think we don't want the mask token id to be one of those in the codebook. Similar thing for the sos token id.

dome272 commented 2 years ago

Oh yea thats correct. My bad. Thanks for correcting. Although it doesnt seem to harm performance that bad, Im currently training another model at 32x32 resolution. Thats the loss curve:

dome272 commented 2 years ago

I restarted training and it seems to perform better now. (Green is the fixed tokens) But I also changed something at the optimiter weight decay, so maybe this also has some effect on it.