Masks for use_face_segmentation_condition

[gordinmitya]

Hello, first of all many thanks for you excellent work, with latest release it really become capable of reconstructing faces!

I have two questions about how masks for use_face_segmentation_condition are computed:

1. mediapipe has selfie_segmentation model (which in fact used in eg meet calls for background replacement) why don't you use this but instead construct rectangles?
2. blur radius seems to be extremely large, isn't it? for example:

is it supposed to be like this?

[cloneofsimo]

Yes, so because they are going to be blurred anyways, I've used just plane face detection module. You can use any sort of pipeline you wish, but It is going to be 8 times compressed (via VAE) + you probably want to learn how face interacts with backgrounds a bit, so I've used blurring in this case. I've found that even blurring isn't enough to learn interaction, so I've added a bias term so that it learns a bit about backgrounds in general. Without it, it makes the whole thing with weird artifacts. But it might be due to overfitting of LoRA, and might be resolved with regularization dataset, but I haven't done those experiments.

Of course, using too much blur makes face conditioning irrelevant, so it is undesirable. So clearly there is a tradeoff.

[cloneofsimo]

Next update will be taking the module out side of dataset, so you can optionally prepare any mask images you want to work from there.

[gordinmitya]

Thank you for such fast answer! Initially I started investigation because I got pretty good results for overall face shape but poor results for eyes/iris region. So I though of slightly increasing loss especially for eyes. I'll update here if my experiment will improve quality.