Open quqixun opened 2 years ago
Thank you for your affection for HifiFace, this project is still undergoing the approval process of open source. Thank you for your patience. Additionally we sincerely hope you can raise your question again in English and these information can be shared by more of this community. Here I will answer your questions. Feel free to keep in touch.
Implementation Details
part of 4. Experiments
For our model with resolution 256 (i.e., Ours-256), we remove images with either size smaller than 256 for better image quality.
The size of Celebrity-Asian
is roughly 680k, while that of 'VGG-Face2' is 640k
Feedbacks for question 3 are listed below.
For questions in part 3,
We generate I
and M
both from feature maps of corresponding size, where for
We do not transform it. The loss is adjusted by parameters.
Your findings are correct. We do not detach I_r and the loss value is relatively small.
For questions in part 3,
Thanks a lot for your response. It is very helpful to understand the paper better.
Some questions about dataset:
Some questions about training:
Some questions about implementation:
About Section 5
About Section 6
About Section 4
For questions in part 3,
- in every mini-batch, we put 50% pairs of same identity and another 50% of different identity.
- In discriminator, we set normalize=False when setting ResBlks, so actually there are no normalization layers in Discriminator.
- You can refer to the supplementary materials of our arxiv version and Questions of SFF #2 issue to find information you need.
- Our HRNet is trained on face data, you can use any face segmentation model that performs well.
- Our entire model is trained end2end and discriminator starts its training from 1st epoch. What to be mentioned is that it may require 1000-5000 iters for the training to warmup, before which the G generates nothing.
- When source face is much thinner than target, it is the most challenging case of face shape preserved swapping. Our SFF is designed to handle this but still not a perfect solution. You are sincerely welcome to discuss your ideas on this issue with us. We are also continually working on bringing out more robust and crazy results.
What is the effect to the generator and the synthesis result whether to use InstanceNorm in Dicriminator? I have this question for a long time. In my own re-implementation, I found the loss of Discriminator always very high without InstanceNorm, and the synthesis is terrible. When I added InstanceNorm, the result is OK. Besides, have you tried to use PatchGAN?
For questions in part 3,
- in every mini-batch, we put 50% pairs of same identity and another 50% of different identity.
- In discriminator, we set normalize=False when setting ResBlks, so actually there are no normalization layers in Discriminator.
- You can refer to the supplementary materials of our arxiv version and Questions of SFF #2 issue to find information you need.
- Our HRNet is trained on face data, you can use any face segmentation model that performs well.
- Our entire model is trained end2end and discriminator starts its training from 1st epoch. What to be mentioned is that it may require 1000-5000 iters for the training to warmup, before which the G generates nothing.
- When source face is much thinner than target, it is the most challenging case of face shape preserved swapping. Our SFF is designed to handle this but still not a perfect solution. You are sincerely welcome to discuss your ideas on this issue with us. We are also continually working on bringing out more robust and crazy results.
What is the effect to the generator and the synthesis result whether to use InstanceNorm in Dicriminator? I have this question for a long time. In my own re-implementation, I found the loss of Discriminator always very high without InstanceNorm, and the synthesis is terrible. When I added InstanceNorm, the result is OK. Besides, have you tried to use PatchGAN?
Sorry for a late reply.
It is quite weird that you encounter this issue. In fact, in our implementation, we do not use instance norm in the discriminator. In the generator, besides AdaIn, we use InstanceNorm in the encode and bottleneck part.
I wonder if your BP of D loss is correct. Remember to detach the generator part when BP the D loss.
For questions in part 3,
- in every mini-batch, we put 50% pairs of same identity and another 50% of different identity.
- In discriminator, we set normalize=False when setting ResBlks, so actually there are no normalization layers in Discriminator.
- You can refer to the supplementary materials of our arxiv version and Questions of SFF #2 issue to find information you need.
- Our HRNet is trained on face data, you can use any face segmentation model that performs well.
- Our entire model is trained end2end and discriminator starts its training from 1st epoch. What to be mentioned is that it may require 1000-5000 iters for the training to warmup, before which the G generates nothing.
- When source face is much thinner than target, it is the most challenging case of face shape preserved swapping. Our SFF is designed to handle this but still not a perfect solution. You are sincerely welcome to discuss your ideas on this issue with us. We are also continually working on bringing out more robust and crazy results.
What is the effect to the generator and the synthesis result whether to use InstanceNorm in Dicriminator? I have this question for a long time. In my own re-implementation, I found the loss of Discriminator always very high without InstanceNorm, and the synthesis is terrible. When I added InstanceNorm, the result is OK. Besides, have you tried to use PatchGAN?
Sorry for a late reply.
It is quite weird that you encounter this issue. In fact, in our implementation, we do not use instance norm in the discriminator. In the generator, besides AdaIn, we use InstanceNorm in the encode and bottleneck part.
I wonder if your BP of D loss is correct. Remember to detach the generator part when BP the D loss.
Yeah, I fixed some tricks and the discriminator without IN worked now, but there seems no difference with the discriminator using IN. I really wonder the effect of IN on the discriminator, and single output v.s. PatchGAN. I'd be very happy to learn your opinion! Besides, I find the synthesis in videos are not as realistic as single images. In the videos, the faces are often suffered from color jittering and shaking between consequtive frames (I already used alpha-filtering in face alignment and color re-normalization). Do you have any suggestions? Thank you very much!
HifiFace is mainly researched on image based swapping instead of video. So it is normal that the model performs not that perfect in video, as we did not tune this.
If you need any help on video synthesis, feel free to drop me an email and maybe we can provide an official generation for you to compare with your result.
HifiFace is mainly researched on image based swapping instead of video. So it is normal that the model performs not that perfect in video, as we did not tune this.
If you need any help on video synthesis, feel free to drop me an email and maybe we can provide an official generation for you to compare with your result.
Hi, I've send an email to hififace.youtu@gmail.com, wish for your reply! Thanks!
Hi, according to what you mentioned above, M_{low}
is generated by z_{dec}
passed through a conv layer with sigmoid act and I_{low}
is generated by z_{dec}
through a lrelu and a conv layer?
Your job is great and really nice. My re-implementation result seems to be OK. But I have some questions about re-implementation.
And in my experimentation, PatchGAN seems to perform better. Hope for your nice answer!
Hi, according to what you mentioned above,
M_{low}
is generated byz_{dec}
passed through a conv layer with sigmoid act andI_{low}
is generated byz_{dec}
through a lrelu and a conv layer?
Exactly
Your job is great and really nice. My re-implementation result seems to be OK. But I have some questions about re-implementation.
- Does the adv or Lpips loss apply on 64x64 level images?
- Is The GAN loss implemented by WGAN-GP?
- Does the adv loss apply on I_cyc?
And in my experimentation, PatchGAN seems to perform better. Hope for your nice answer!
Thank you very much for your advice, we will definitely try more SOTA backbones for better results!
Thx! I still have some questions:
Thx! I still have some questions:
- Face model generated with coeff in loss Lshape, does q_fuse (or qr) stand for face_shape in these lines of codes? https://github.com/sicxu/Deep3DFaceRecon_pytorch/blob/master/models/bfm.py#L86-L99
- You mentioned lpips loss makes sense only with two images of same person, so all data paired with the same person in the training set (source and target is the same person)?
landmark
stands for those 17 facial contour points among 68 landmark points. fused embedding
stands for an embedding that uses the source's shape id, and target's expr+pose id. The reconstruction process does not use textures.Hi, author, thanks your great job! I have some questions about implementation details:
I try to re-implement the paper,but meet some troubles.Ask for help. First,after training for a long time, the mask tend to all empty(all 0).rec loss stay near 0. To fix that,i just use segmap loss , I find that high segLoss can work well but the low segloss is invalid. i think the segHigh tend to force the segLow to empty for a better effect. I think it establish a short path for high loss and longer and more difficult path for the low one.
For questions in part 3,
- in every mini-batch, we put 50% pairs of same identity and another 50% of different identity.
- In discriminator, we set normalize=False when setting ResBlks, so actually there are no normalization layers in Discriminator.
- You can refer to the supplementary materials of our arxiv version and Questions of SFF #2 issue to find information you need.
- Our HRNet is trained on face data, you can use any face segmentation model that performs well.
- Our entire model is trained end2end and discriminator starts its training from 1st epoch. What to be mentioned is that it may require 1000-5000 iters for the training to warmup, before which the G generates nothing.
- When source face is much thinner than target, it is the most challenging case of face shape preserved swapping. Our SFF is designed to handle this but still not a perfect solution. You are sincerely welcome to discuss your ideas on this issue with us. We are also continually working on bringing out more robust and crazy results.
What is the effect to the generator and the synthesis result whether to use InstanceNorm in Dicriminator? I have this question for a long time. In my own re-implementation, I found the loss of Discriminator always very high without InstanceNorm, and the synthesis is terrible. When I added InstanceNorm, the result is OK. Besides, have you tried to use PatchGAN?
Sorry for a late reply. It is quite weird that you encounter this issue. In fact, in our implementation, we do not use instance norm in the discriminator. In the generator, besides AdaIn, we use InstanceNorm in the encode and bottleneck part. I wonder if your BP of D loss is correct. Remember to detach the generator part when BP the D loss.
Yeah, I fixed some tricks and the discriminator without IN worked now, but there seems no difference with the discriminator using IN. I really wonder the effect of IN on the discriminator, and single output v.s. PatchGAN. I'd be very happy to learn your opinion! Besides, I find the synthesis in videos are not as realistic as single images. In the videos, the faces are often suffered from color jittering and shaking between consequtive frames (I already used alpha-filtering in face alignment and color re-normalization). Do you have any suggestions? Thank you very much!
How did you apply color re-normalization? Is there any reference article or code?
您好,我在复现您的这篇论文,在复现过程中遇到了以下问题:
从数据集Celebrity-Asian和VggFace2中裁剪人脸,存在很多模糊人脸,请问数据集中的模糊数据您是如何处理的,您最终用于训练模型的数据量大概是多少,这部分在论文中没有详细提及;
在Feature-Level最后一段中:
在Experiments中:
再结合您在issue2中的回答,您看我下面的理解是否正确: 2.1 Ours-256 和 Ours-512 模型输入的 It 均为256尺寸图像; 2.2 在 Ours-256 模型中,在获得 zfuse 后,先过两层 AdaIN Res-Block,再过两层有upsample的Res-Block的Fup; 2.3 Ours-512 与 Ours-256 相比,区别仅在于,在Ours-512模型的 Fup模块中多一层有upsample的Res-Block;
其他问题: 3.1 获得 Mlow 、Mr、Ilow、Ir过程中,输出层结构是怎样的; 3.2 使用3D人脸模型获得人脸关键点后,其坐标范围为[0, 224],是否将其转换为[0, 1]范围; 3.3 在计算 Lcyc 时,cyc过程输出G(Ir , It),Ir是否有detach;训练过程中发现Lcyc 特别低,相比其他loss差一到两个数量级; 3.4 训练过程中是怎样做数据采样的; 3.5 在辨别器中使用的也是Res-Block,Res-Block中使用的是InstanceNorm2D,这里请您确认在辨别器中使用的是InstanceNorm2D; 3.6 能否列举SFF的详细结构; 3.7 HRNet的人脸分割效果比较差,是否做了其他优化; 3.8 是否是分阶段训练的,还是在一开始辨别器就参与了训练; 3.9 脸型差异较大时,生成结果存在双下巴现象,这种现象是否是靠辨别器抑制掉的。
问题较多,期待回复,谢谢。