Can't load model when I train transformer after training pvqvae

[zhougeAI]

I use pvqvae_imagenet.yaml and transformer_imagenet.yaml to train on my own dataset. Training pvqvae is OK, but when I train transformer, I meet a model loading problem: RuntimeError: Error(s) in loading state_dict for MaskedImageInpaintingTransformer: Missing key(s) in state_dict: "pos_emb", "content_codec.encoder.pre_layers.0.weight", "content_codec.encoder.pre_layers.0.bias", "content_codec.encoder.res_layers.0.layers.1.weight", "content_codec.encoder.res_layers.0.layers.1.bias", ...... Unexpected key(s) in state_dict: "encoder.pre_layers.0.weight", "encoder.pre_layers.0.bias", "encoder.res_layers.0.layers.1.weight", "encoder.res_layers.0.layers.1.bias", "encoder.res_layers.0.layers.3.weight", "encoder.res_layers.0.layers.3.bias", "encoder.res_layers.1.layers.1.weight", "encoder.res_layers.1.layers.1.bias", "encoder.res_layers.1.layers.3.weight", ..... How to solve it?

[liuqk3]

Hi @zhougeAI ,

You should provid the path of pvqvae model weights in the configure file of transformer, then it will be automatically loaded. You can refer to this line for example:

https://github.com/liuqk3/PUT/blob/7de8ce0ada1e63e8c5300857a463d68380e142f0/configs/put_cvpr2022/ffhq/transformer_ffhq.yaml#L23