About the training loss

[hanssssssss]

Hi ! I re-implemented the training process using pytorch, but I encounteered some problems durning training.1. The landmark loss was vert high even though I use the loss weight(1.6e-3) to constrain it. 2.I used the pretrained resnet , but during first 10 epoch of training, the photomatic loss is 0, I had no idea why these happen.

Thanks,

This is the statistics of the training.

[YuDeng]

We set the weights and bias of the last fc layer to zero at the beginning, so that initially the rendered image should contain a frontal face in the center. I guess the zero photometric loss and large landmark loss is caused by arbitrary network outputs at the very first iteration. You may check if the rendered face falls inside the image in your case.

[hanssssssss]

We set the weights and bias of the last fc layer to zero at the beginning, so that initially the rendered image should contain a frontal face in the center. I guess the zero photometric loss and large landmark loss is caused by arbitrary network outputs at the very first iteration. You may check if the rendered face falls inside the image in your case.

Thanks !!! Actually, the weights and bias are initialised by built-in function(they aren't zero), and the I couldn't see any face in the render image. I will reset the weights and bias .

I sincerely appreciate your help !

[steventai0212]

@hanssssssss hi! Can you share your pytorch re-implemented code? Thanks a lot!

[hanssssssss]

@hanssssssss hi! Can you share your pytorch re-implemented code? Thanks a lot!

Hi! I am sorry, I haven't finished the code yet. Once I complete the code, I will share it with you

[hanssssssss]

We set the weights and bias of the last fc layer to zero at the beginning, so that initially the rendered image should contain a frontal face in the center. I guess the zero photometric loss and large landmark loss is caused by arbitrary network outputs at the very first iteration. You may check if the rendered face falls inside the image in your case.

Hi ! Sorry for bothering you. I reseted the weights and trained the model successfully. But I encounteered the mean face problem. I removed the regularization loss as you mentioned in the previous issues, but the reconstruction was still a mean face. Do you have any idea about it ?

Thanks.

This is the face with regularization loss.

This is the face without regularization loss.

[YuDeng]

There are many problems that might cause a learned face similar to the mean face. I think maybe you can try to overfit a single image (train the pipeline with single image) to see what happens. The network should produce reasonable result if everything is correct. And you can train the network with more iterations to check if the result becomes better.

[hanssssssss]

There are many problems that might cause a learned face similar to the mean face. I think maybe you can try to overfit a single image (train the pipeline with single image) to see what happens. The network should produce reasonable result if everything is correct. And you can train the network with more iterations to check if the result becomes better.

Thanks for your help! I will try it now.

[hanssssssss]

There are many problems that might cause a learned face similar to the mean face. I think maybe you can try to overfit a single image (train the pipeline with single image) to see what happens. The network should produce reasonable result if everything is correct. And you can train the network with more iterations to check if the result becomes better.

Hi ! Sorry for bothering you again. I have tried many combinations of values of weights to overfit a single image(I tried 1000 iteration for a single image). But I found that the color and the texture of the face was bad(and the render image of each model looks the same ). I also added the gamma loss when training the model, but the gamma loss was always 0 even thoung I added a small values(1e-4) when caculating it. Do you have any idea about the issues ? I had no idea about it.Should I tried more combinations of weights ?

Thanks for your help.

overfitting image

W_photo = 1.92 W_landmark = 1.6e-3 W_perceptual = 0.05 W_reg = 1.5e-4 W_ref = 3 W_gamma = 10.0

W_photo = 1.92 W_landmark = 1.6e-3 W_perceptual = 0.05 W_reg = 3.0e-4 W_ref = 5.0 W_gamma = 10.0

The loss of the image seems reasonable, but the render image and 3d face are bad

photo_loss tensor(0.2203, device='cuda:0') perceptual_loss tensor(0.0014, device='cuda:0') landmark_loss tensor(0.0660, device='cuda:0') reg_loss tensor(0.0039, device='cuda:0') ref_loss tensor(0.0225, device='cuda:0') gamma_loss tensor(1.0000e-07, device='cuda:0')

[YuDeng]

Seems that the weight of the perceptual loss is too small. In addition, what network did you use to calculate the perceptual loss?

[hanssssssss]

Seems that the weight of the perceptual loss is too small. In addition, what network did you use to calculate the perceptual loss?

Hi ! I used InceptionResnetV1 from facenet_pytorch('trained on casia-webface') which is similiar to the net in your work. But Its a 512 dimensions output since I just removed the last fc layer.

class FacialFeatureExtractor(InceptionResnetV1): def forward(self, x): x = self.conv2d_1a(x) x = self.conv2d_2a(x) x = self.conv2d_2b(x) x = self.maxpool_3a(x) x = self.conv2d_3b(x) x = self.conv2d_4a(x) x = self.conv2d_4b(x) x = self.repeat_1(x) x = self.mixed_6a(x) x = self.repeat_2(x) x = self.mixed_7a(x) x = self.repeat_3(x) x = self.block8(x) x = self.avgpool_1a(x) x = self.dropout(x) x = self.last_linear(x.view(x.shape[0], -1)) x = self.last_bn(x)

    return x

[YuDeng]

Okay, the network seems reasonable. Then maybe you could try to use a larger weight for perceptual loss.

[hanssssssss]

Okay, the network seems reasonable. Then maybe you could try to use a larger weight for perceptual loss.

Thanks for your help. I will try it now. By the way, Is it necessary to add a gamma loss while training?I got a 3d face like this, I think the color of the face is quite dark compared with the output of your pretrained model. Do the 0 value of gamma loss durning training has something to do with this? Thanks again.

[hanssssssss]

Okay, the network seems reasonable. Then maybe you could try to use a larger weight for perceptual loss.

Hi! Sorry for bothering you for so many times. I adjusted the weight of the perceptual loss from 0.2 to 1.44. At the same time, I found that you frozen the first part of the resnet in your work and I did the same, but I got the similiar 3d face as before.I think the color of the face is too dark. Do you have any idea about it?

Thanks.

This is my weight of loss. W_photo = 1.92 W_landmark = 1.6e-3 W_perceptual = 1.44 W_reg = 5.0e-4 W_ref = 8.0 W_gamma = 10.0

This is the loss of face. photo_loss tensor(0.2221, device='cuda:0') perceptual_loss tensor(0.0205, device='cuda:0') landmark_loss tensor(0.0114, device='cuda:0') reg_loss tensor(0.0143, device='cuda:0') ref_loss tensor(0.0264, device='cuda:0') gamma_loss tensor(1.0000e-05, device='cuda:0')

I also try to add 0.01 when caculating the gamma loss to make the gamma loss a reasonable value(its tensor(0.0016, device='cuda:0')). But the output of the 3d face was strange.

[hanssssssss]

Okay, the network seems reasonable. Then maybe you could try to use a larger weight for perceptual loss.

Hi , sorry for bothering you. After overfitting the single image, I chose reasonable weights of loss to train the model on a batch data. But the gamma loss was always 0, I didn't know why. Do you have any idea about it ? Is it a usual phenomenon of setting the initial weights and bias as 0？

Thanks a lot.

[CircleRina]

@hanssssssss hi! Can you share your pytorch re-implemented code? Thanks a lot!

Hi! I am sorry, I haven't finished the code yet. Once I complete the code, I will share it with you

Hi! Sorry for commenting this closed issue. :flushed: Have you finished this code now?:smiley: Thanks for your help!