The value of the loss function increase during the training

[abdou31]

Hello Yin, As you know for the first model that I have created with this CNN, it worked perfectly but for the face oriented ( profile face), the tracking is lost and I have discovered that the problem is coming from the false annotations. After some research, I have found a solution, So I have organized a new dataset and I have annotated this dataset using OpenFace, for now, the dataset is correctly annotated. I followed all the steps to create the three files validation.record, train.record and test.record I have started the training, after some steps (60 000), I stop the training and I have seen the graph of the loss function using Tensorboard. The result was so bad as you can see:

comparing to the old model:

I would like what is the problem here? How can I correct the accuracy of the prediction?

Note:

• I have used the old version of CNN facial landmark project
• I have added some other dataset to ibug dataset ( the number of images is 250 000)

[yinguobing]

That's hard to tell. Try reducing the learning rate first.

[abdou31]

What value will be good for learning_rate? What if I set it for 0.0001 ?

[abdou31]

After reducing the learning_rate and re-split the dataset random( give 140000 images to train instead of 150000 images). I visualize the loss graph and that was the result:

But when I test on video; I get so bad results (I can say that there is no result ) compare to the other model, I don't know why the results are so different to the graph loss and to the learning_rate???

This is really so stranger

I doubt that the problems coming from the dataset, not anything else, anyway I have used the same dataset ( 300VW ) in the two situations. How can I fix that?

Update:

As I have seen the result, I have two things that can be the source of the problem:

• the first one is before extracting the face from the ibug dataset using extract_face_from_ibug.py, I have changed the value of the preview_face_view from 512 to 300 on face_detector.py, and that maybe can give a bad training with a false face detection of the dataset ( beacause I have seen some images of this dataset that the face is bad detected ).
• The second thing is that I have drawn the pts on images of the dataset ( and this may be made the dataset be not good for the training ).

For that reason, I will try to correct those things and I will see the results, if you have an idea or a suggestion to solve this problem, please let me know. Thanks.

[abdou31]

I solved the problem by following the two points that I have mentioned in the previous comment.

[ZlaaM]

@abdou31 Is the new dataset you used 68 landmarks? if yes please I need it

[abdou31]

No I used the same dataset but with different number of annotation landmarks

[ZlaaM]

@abdou31 thank you

[ZlaaM]

@abdou31 How much value of train steps and learning rate do I have to choose to get good results? do you have an idea? thanks

[abdou31]

@ZlaaM
I have set 150 000 steps and for learning_rate=1e-3