A confusing question in transformer training

[forever-rz]

Thanks for your contribution,but there is a problem when i train it on FFHQ.Once the ratio of mask is larger, it seems that only part of the completed result is repaired, the part that is not repaired stays black and no new content seems to be generated. Is this normal? eg1: First on the left, second on the left remain some strange black regions .(epoch:12) mask_input completed reconstruction

[forever-rz]

This bad visual effect became increasingly apparent later in the training. eg2: epoch:19 mask_input completed reconstruction input

[liuqk3]

Hi @forever-rz , Thanks for your interests. It seems that the second codebook is used for quantization while training transformer. For FFHQ, it will not take so long to get a reasonable inpainting results.

[forever-rz]

@liuqk3 Thanks for your help! Can you tell me how should I use just one codebook ? I only modified three parts of the transformer training configuration( batch_size,sample_iterations,save_epochs), the rest was followed by the pre-trained model configuration.This is really confusing to me.

[forever-rz]

My configuration is shown below： dataloader: batch_size: 16 data_root: data num_workers: 1 train_datasets:

• params: im_preprocessor_config: params: horizon_flip: true random_crop: true size:
  • 256
  • 256 smallest_max_size: 256 target: image_synthesis.data.utils.image_preprocessor.SimplePreprocessor image_list_file: data/ffhqtrain_69k.txt mask: 1.0 mask_low_size:
    • 32
    • 32 mask_low_to_high: 0.1 multi_image_mask: false name: ffhq provided_mask_name: irregular-mask/testing_mask_dataset return_data_keys:
    • image
    • mask stroken_mask_params: keep_ratio:
    • 0.3
    • 0.6 maxBrushWidth: 30 maxLength: 100 maxVertex: 10 minBrushWidth: 10 minVertex: 5 min_area: 64 use_provided_mask: 0.8 use_provided_mask_ratio:
    • 0.3333333
    • 1.0 target: image_synthesis.data.image_list_dataset.ImageListDataset validation_datasets:
• params: im_preprocessor_config: params: size:
  • 256
  • 256 smallest_max_size: 256 target: image_synthesis.data.utils.image_preprocessor.SimplePreprocessor image_list_file: data/ffhqvalidation_1k.txt mask: 1.0 mask_low_size:
    • 32
    • 32 mask_low_to_high: 0.0 multi_image_mask: false name: ffhq provided_mask_name: irregular-mask/testing_mask_dataset return_data_keys:
    • image
    • mask
    • relative_path stroken_mask_params: keep_ratio:
    • 0.3
    • 0.6 maxBrushWidth: 30 maxLength: 100 maxVertex: 10 minBrushWidth: 10 minVertex: 5 min_area: 64 use_provided_mask: 1.0 #TODO0.8 use_provided_mask_ratio:
    • 0.4
    • 1.0 target: image_synthesis.data.image_list_dataset.ImageListDataset

model: target: image_synthesis.modeling.models.masked_image_inpainting_transformer_in_feature.MaskedImageInpaintingTransformer params: n_layer: 30 content_seq_len: 1024 n_embd: 512 n_head: 8 num_token: 512 embd_pdrop: 0.0 attn_pdrop: 0.0 resid_pdrop: 0.0 attn_content_with_mask: False mlp_hidden_times: 4 block_activate: GELU2 random_quantize: 0.3 weight_decay: 0.01 content_codec_config: target: image_synthesis.modeling.codecs.image_codec.patch_vqgan.PatchVQGAN params: ckpt_path: OUTPUT/pvqvae_ffhq/checkpoint/last.pth trainable: False token_shape: [32, 32] combine_rec_and_gt: True quantizer_config: target: image_synthesis.modeling.codecs.image_codec.patch_vqgan.VectorQuantizer params: n_e: 1024 e_dim: 256 masked_embed_start: 512 embed_ema: True get_embed_type: retrive distance_type: euclidean encoder_config: target: image_synthesis.modeling.codecs.image_codec.patch_vqgan.PatchEncoder2 params: in_ch: 3 res_ch: 256 out_ch: 256 num_res_block: 8 res_block_bottleneck: 2 stride: 8 decoder_config: target: image_synthesis.modeling.codecs.image_codec.patch_vqgan.PatchConvDecoder2 params: in_ch: 256 out_ch: 3 res_ch: 256 num_res_block: 8 res_block_bottleneck: 2 stride: 8 up_layer_with_image: true encoder_downsample_layer: conv solver: adjust_lr: none base_lr: 0.0 find_unused_parameters: false max_epochs: 250 optimizers_and_schedulers:

• name: transformer optimizer: params: betas: !!python/tuple
  • 0.9
  • 0.95 target: torch.optim.AdamW scheduler: params: min_lr: 1.0e-05 warmup: 2000 warmup_lr: 0.0003 step_iteration: 1 target: image_synthesis.engine.lr_scheduler.CosineAnnealingLRWithWarmup sample_iterations: 400 save_epochs: 1 validation_epochs: 1

[forever-rz]

The strange thing is that when the mask ratio is small, like in the green circle, there are no such problems and only one codebook seems to be used, so why is it possible that two codebooks are used when the mask ratio is large (in the red circle)? What's wrong with my settings? eg epoch:8 mask_input completed

[forever-rz]

@liuqk3 Thanks for the reply, but I've carefully compared the posted model to my model training process and really don't notice any difference. So was wondering if it could be the parameters? In the training phase of pvqvae keep_ratio:[0.0,0.5] while the transformer training phase keep_ratio:[0.3,0.6] causes this?

[liuqk3]

Hi, @forever-rz . Sorry for the delayed reply.

keep_ratio just affects the number of remained pixels in an image, it should not cause such as artifacts. After have a loot at your configs, I do not find something wrong. Here is my questions or suggestions: 1) Did you use our provided P-VQVAE or train it by yourself? 2) Have you checked the reconstruction capability of the used P-VQVAE? 3) Can you provide the cross-entropy loss curves of transformer?

[forever-rz]

Hi, @liuqk3 .I'm sorry that I temporarily put aside the experiment because I couldn't figure out the cause of this problem. Today, I carefully checked the previous experiment and found the relevant data of the three questions you raised as follows. 1 Instead of using the provided P-VQVAE, I trained a new P-VQVAE model companion and added some attention blocks. 2 The reconstruction results of my PVQVAE model are as follows FFHQ (a)input (b) mask (c) reference_input (d)reconstruction

PLACES2 (a)input (b) mask (c) reference_input (d)reconstruction IMAGENT (a)input (b) mask (c) reference_input (d)reconstruction

3 The loss of training transformer based on my P-VQVAE is as follows (ImageNet is too big, so it is given up) FFHQ: Places2

[forever-rz]

It seems to me that the reconstruction results are not bad, so I don't understand why transformer's training results are so wrong. Although I made some changes to P-VQVAE, transformer has not changed at all.

[liuqk3]

@forever-rz ， I do not know the number of epochs you have trained on FFHQ and Places2. You can try to visualize the inpainting results of the trained model.

[myhansu]

Hi, do you have any updates on this issue? I have also encountered same problem with a custom dataset, reconstruction results are much better than completed results.

[UESTC-Med424-JYX]

I also encountered the problem of good reconstruction effect but poor generation effect in a similar task. I saw that our loss curves are basically the same. Did you solve this problem in the end?