How do I preserve the background and fix the mask position?

[Chaejongwook1]

Hello. Thank you for providing a good source code.

I am a university student who is doing research in Korea. Currently, I want to create a polyp in the mask part by synthesizing a mask in the image with pixel2style2pixel. Below are the training result and the inference result.

training result

inference result

What I want here is to preserve the background as it is and create a polyp in the mask, but it's not being done properly. What I want the most is to produce a polyp exactly at the location of the mask.

Is it impossible to work with the model I want? If possible, I would appreciate it if you could advise me in which direction to proceed with the learning.

polyp_inpainting 2022-09-26

!python scripts/train.py \ --dataset_type=ffhq_encode \ --exp_dir=experiment/{date()} \ --workers=8 \ --batch_size=16 \ --test_batch_size=16 \ --test_workers=8 \ --val_interval=2500 \ --save_interval=5000 \ --encoder_type=GradualStyleEncoder \ --start_from_latent_avg \ --lpips_lambda=0.8 \ --l2_lambda=2 \ --moco_lambda=0.2 \ --use_wandb \ --output_size=256 \ --stylegan_weights="pretrained_models/network-snapshot-000800.pt"

[yuval-alaluf]

If you are seeing that the background is not preserved, you can create the final output image by taking the non-masked region from the original image and the masked region from your generated result. Then, you will need to do some post-processing to merge the two regions together as they will not look natural if you combine them naively. You could try applying a poison blending or some soft mask blending between the two regions. Hope this helps.

[Chaejongwook1]

Thank you for your answer. Is there a way to induce polyps through hyperparameters so that they are accurately produced in the mask position even if the quality is poor? Since the mask position should be used as Ground Truth, we want a polyp to be generated accurately in the white part even if the quality is poor.

[yuval-alaluf]

You could possibly try to strengthen the loss within the masked region. This could force the encoder to focus more on the area of interest in your case.