Region editing

[kudoNCT]

Hi @taesungp , great project ! I have a question that in your paper and demo video you show that the model can change the region in the object using PCA collaborate with the mask user draw. Can you explain more detail. Thank you!

[taesungp]

It was based on running PCA on the structure code.

For the landscape editing, we collected the structure codes across all locations and images, and then run PCA. In more detail, I randomly chose 5000 images from the training set and extracted the structure codes to produce a N x C x H x W tensor, where N=5000, HW is the feature resolution of the images, and C the dimension of the structure code. I reshape it to NHW x C tensor, and run PCA. Therefore, each principal axis of PCA is a C-dimensional vector. In the UI, I just add the same manipulation vector to the selected region. Additionally, I automatically created the mask by computing a self-similarity map of the clicked region based on the cosine similarity of the structure codes. But this is optional.

For face editing, because the faces are aligned, we flattened the codes to HWC-dimensional vector and then run PCA on it. Note that it only makes sense because the facial landmarks are aligned in the FFHQ dataset.

[kudoNCT]

Hi @taesungp , thanks for your reply. Because I am a newbie so can you explain why we need use PCA here and how we get predefined manipulation vector with PCA in face editing? 1 manipulation vector with 1 image? I see n_dimension PCA in your code is set default 128 dims. Does it also is 128 in face editing and landscape editing ? .Thanks a lot for your helping me to understand more !

[hughwcq]

Hi,@kudoNCT, may I ask you where the code of PCA is? Why I don`t find it? And I think PCA can find the main attributes of the images in the dataset, the predefined manipulation vectors need to be calculated by many images. In detail, you can refer to this paper: GANspace.

[kudoNCT]

Hi @hughwcq , thanks for your suggest. I find the code of PCA in util directory.