search

nii-yamagishilab / ClassNSeg

Implementation and demonstration of the paper: Multi-task Learning for Detecting and Segmenting Manipulated Facial Images and Videos
BSD 3-Clause "New" or "Revised" License
78 stars 12 forks source link

labels influence segments results #4

Open dong03 opened 3 years ago

dong03 commented 3 years ago

https://github.com/nii-yamagishilab/ClassNSeg/blob/87e506fddb9f36ef14f9bd1f6496f86d7faef0fd/train.py#L382

As is shown in #L382, labels participant in segmentation. When I change label from [1,0] to [0,1], the real_seg becomes a meaningful mask and the fake_seg appears to be all black. It is exactly the opposite.

Whether the segmentation able to find the border in fake image？

dong03 commented 3 years ago

https://github.com/nii-yamagishilab/ClassNSeg/blob/87e506fddb9f36ef14f9bd1f6496f86d7faef0fd/train.py#L363

（change labels here

honghuy127 commented 3 years ago

That's an interesting question. I have a theory about this.

The network focuses only on facial region. The activation of the latent feature measures how certain it believe the facial region in this mask was manipulated. When you force the label to the opposite direction, these low activation latents are reversed, resulting in some kind of segmentation mask in the output.

Since in normal cases, the segmentation masks somehow have the shape of the ground-truth, I believe that the segmentation branch is able to find the border.