The Value of the Corr is nan

[Cherry2410]

Hi, In the training of CEN, the value of the Corr turns into nan. The datasets of Helen and LFPW are used. I do not understand this phenomenon. Can you give some advice?

[ghost]

Hello. It should not turn into Nan if you are running the hyperparameters assigned in the training file. Can you tell us more about what is the value of MSE? Have you created the training data from the provided script?

[ghost]

A small thing to consider is using Arch4 from the architectures we have designed. That is the architecture we used for our released files.

[Cherry2410]

@A2Zadeh, here are some training results. Landmark 8 Train on 1800225 samples, validate on 200070 samples Epoch 1/100

• 122s - loss: 0.2716 - acc: 0.3852 - val_loss: 0.2026 - val_acc: 0.4453 RMSE: 0.4500953403005947 Corr: 0.0009191826686431913 Epoch 00001: saving model to D:\openface-training\cen_training./model_saves\model_half\0.25_profile1_8_256\general_epoch_1.h5 Epoch 2/100
• 115s - loss: 0.1850 - acc: 0.4987 - val_loss: 0.1673 - val_acc: 0.4453 RMSE: 0.40901706271222055 Corr: 0.0007946218698609128 Epoch 00002: saving model to D:\openface-training\cen_training./model_saves\model_half\0.25_profile1_8_256\general_epoch_2.h5 Epoch 3/100
• 114s - loss: 0.1519 - acc: 0.4987 - val_loss: 0.1366 - val_acc: 0.4453 RMSE: 0.36957058766795337 Corr: 0.0007388850332561021 Epoch 00003: saving model to D:\openface-training\cen_training./model_saves\model_half\0.25_profile1_8_256\general_epoch_3.h5 Epoch 4/100
• 116s - loss: 0.1234 - acc: 0.4987 - val_loss: 0.1103 - val_acc: 0.4453 RMSE: 0.3321888304344669 Corr: 0.00048061614529095386 Epoch 00004: saving model to D:\openface-training\cen_training./model_saves\model_half\0.25_profile1_8_256\general_epoch_4.h5 Epoch 5/100
• 113s - loss: 0.0992 - acc: 0.4987 - val_loss: 0.0884 - val_acc: 0.4453 RMSE: 0.29725507962419884 Corr: 3.4957091515424905e-07 Epoch 00005: saving model to D:\openface-training\cen_training./model_saves\model_half\0.25_profile1_8_256\general_epoch_5.h5 Epoch 6/100
• 115s - loss: 0.0791 - acc: 0.4987 - val_loss: 0.0703 - val_acc: 0.4453 RMSE: 0.26505373734603843 Corr: nan Epoch 00006: saving model to D:\openface-training\cen_training./model_saves\model_half\0.25_profile1_8_256\general_epoch_6.h5 Epoch 7/100
• 113s - loss: 0.0627 - acc: 0.4987 - val_loss: 0.0556 - val_acc: 0.4453 RMSE: 0.2357599985765704 Corr: nan Epoch 00007: saving model to D:\openface-training\cen_training./model_saves\model_half\0.25_profile1_8_256\general_epoch_7.h5 Epoch 8/100
• 115s - loss: 0.0495 - acc: 0.4987 - val_loss: 0.0439 - val_acc: 0.4453 RMSE: 0.20946750451663806 Corr: nan Epoch 00008: saving model to D:\openface-training\cen_training./model_saves\model_half\0.25_profile1_8_256\general_epoch_8.h5 Epoch 9/100
• 113s - loss: 0.0390 - acc: 0.4987 - val_loss: 0.0347 - val_acc: 0.4453 RMSE: 0.1861909687652356 Corr: nan Epoch 00009: saving model to D:\openface-training\cen_training./model_saves\model_half\0.25_profile1_8_256\general_epoch_9.h5 Epoch 10/100
• 112s - loss: 0.0308 - acc: 0.4987 - val_loss: 0.0275 - val_acc: 0.4453 RMSE: 0.16586733331679024 Corr: nan Epoch 00010: saving model to D:\openface-training\cen_training./model_saves\model_half\0.25_profile1_8_256\general_epoch_10.h5 Epoch 11/100
• 114s - loss: 0.0244 - acc: 0.4987 - val_loss: 0.0220 - val_acc: 0.4453 RMSE: 0.14839784523929922 Corr: nan Epoch 00011: saving model to D:\openface-training\cen_training./model_saves\model_half\0.25_profile1_8_256\general_epoch_11.h5 Epoch 12/100
• 113s - loss: 0.0196 - acc: 0.4987 - val_loss: 0.0179 - val_acc: 0.4453 RMSE: 0.13361902832765732 Corr: nan Epoch 00012: saving model to D:\openface-training\cen_training./model_saves\model_half\0.25_profile1_8_256\general_epoch_12.h5 Epoch 13/100
• 114s - loss: 0.0159 - acc: 0.4987 - val_loss: 0.0147 - val_acc: 0.4453 RMSE: 0.12134974950293488 Corr: nan Epoch 00013: saving model to D:\openface-training\cen_training./model_saves\model_half\0.25_profile1_8_256\general_epoch_13.h5 Epoch 14/100
• 114s - loss: 0.0132 - acc: 0.4987 - val_loss: 0.0124 - val_acc: 0.4453 RMSE: 0.11135111945295995 Corr: nan Epoch 00014: saving model to D:\openface-training\cen_training./model_saves\model_half\0.25_profile1_8_256\general_epoch_14.h5 Epoch 15/100
• 114s - loss: 0.0111 - acc: 0.4987 - val_loss: 0.0107 - val_acc: 0.4453 RMSE: 0.10338035679713743 Corr: nan Epoch 00015: saving model to D:\openface-training\cen_training./model_saves\model_half\0.25_profile1_8_256\general_epoch_15.h5 Epoch 16/100
• 113s - loss: 0.0096 - acc: 0.4987 - val_loss: 0.0094 - val_acc: 0.4453 RMSE: 0.09716219400822464 Corr: nan Epoch 00016: saving model to D:\openface-training\cen_training./model_saves\model_half\0.25_profile1_8_256\general_epoch_16.h5 Epoch 17/100
• 113s - loss: 0.0086 - acc: 0.4987 - val_loss: 0.0085 - val_acc: 0.4453 RMSE: 0.09242534320694404 Corr: nan Epoch 00017: saving model to D:\openface-training\cen_training./model_saves\model_half\0.25_profile1_8_256\general_epoch_17.h5 Epoch 18/100
• 115s - loss: 0.0078 - acc: 0.4987 - val_loss: 0.0079 - val_acc: 0.4453 RMSE: 0.08890215716593411 Corr: nan Epoch 00018: saving model to D:\openface-training\cen_training./model_saves\model_half\0.25_profile1_8_256\general_epoch_18.h5 Epoch 19/100
• 111s - loss: 0.0072 - acc: 0.4987 - val_loss: 0.0075 - val_acc: 0.4453 RMSE: 0.08633810614861614 Corr: nan Epoch 00019: saving model to D:\openface-training\cen_training./model_saves\model_half\0.25_profile1_8_256\general_epoch_19.h5 Epoch 20/100
• 113s - loss: 0.0068 - acc: 0.4987 - val_loss: 0.0071 - val_acc: 0.4453 RMSE: 0.08452469765819146 Corr: nan Epoch 00020: saving model to D:\openface-training\cen_training./model_saves\model_half\0.25_profile1_8_256\general_epoch_20.h5 Epoch 21/100
• 114s - loss: 0.0066 - acc: 0.4987 - val_loss: 0.0069 - val_acc: 0.4453 RMSE: 0.08327481402970786 Corr: nan Epoch 00021: saving model to D:\openface-training\cen_training./model_saves\model_half\0.25_profile1_8_256\general_epoch_21.h5 Epoch 22/100
• 115s - loss: 0.0064 - acc: 0.4987 - val_loss: 0.0068 - val_acc: 0.4453 RMSE: 0.08244106386845919 Corr: nan Epoch 00022: saving model to D:\openface-training\cen_training./model_saves\model_half\0.25_profile1_8_256\general_epoch_22.h5 Epoch 23/100
• 116s - loss: 0.0063 - acc: 0.4987 - val_loss: 0.0067 - val_acc: 0.4453 RMSE: 0.08190876809631807 Corr: nan Epoch 00023: saving model to D:\openface-training\cen_training./model_saves\model_half\0.25_profile1_8_256\general_epoch_23.h5 Epoch 24/100
• 112s - loss: 0.0062 - acc: 0.4987 - val_loss: 0.0067 - val_acc: 0.4453 RMSE: 0.08158704925029242 Corr: 3.82595387952076e-06 Epoch 00024: saving model to D:\openface-training\cen_training./model_saves\model_half\0.25_profile1_8_256\general_epoch_24.h5 Epoch 25/100
• 114s - loss: 0.0061 - acc: 0.4987 - val_loss: 0.0066 - val_acc: 0.4453 RMSE: 0.08141099520610463 Corr: nan Epoch 00025: saving model to D:\openface-training\cen_training./model_saves\model_half\0.25_profile1_8_256\general_epoch_25.h5 Epoch 26/100
• 114s - loss: 0.0061 - acc: 0.4987 - val_loss: 0.0066 - val_acc: 0.4453 RMSE: 0.08132777128409081 Corr: nan Epoch 00026: saving model to D:\openface-training\cen_training./model_saves\model_half\0.25_profile1_8_256\general_epoch_26.h5 Epoch 27/100
• 115s - loss: 0.0061 - acc: 0.4987 - val_loss: 0.0066 - val_acc: 0.4453 RMSE: 0.08129907182684672 Corr: 5.0516491000678174e-05

Can you give me some suggestions？ thx.

[Cherry2410]

From the results, it‘s overfitting. And the dataset of afw, Helen, ibug, lfpw, 300w are used.

[ghost]

Thanks @LingQiu. Is this on architecture 4?

[Cherry2410]

@A2Zadeh, we tried the model of arch4 and model_half. Perhaps, every dataset should have its own characteristics.

[ghost]

@LingQiu if you are training on our data you may get nan values but you will recover from it if you continue the training. We basically do MSE optimization and use corr as a measure of visualization rather than direct optimization. Are you able to continue your training and see if corr recovers?

[Cherry2410]

We have been training. The value of MSE is better. Although, the value of the corr can recover from nan, its value is very small. Is this normal?

[ghost]

@LingQiu depending on the landmark number, yes quite possible. Anything higher than 0 is good for some very hard landmarks such as markers around the face. They are hard to detect and disambiguate.

[Cherry2410]

ok,thx.@A2Zadeh

[MoreyLiu]

hello，@LingQiu @A2Zadeh In my process of training cen, I also encountered the corr value of nan. My database is different from the author's (all near infrared images). My training parameters are: num_epochs 100 (generally more than 20 corr value is nan), minibatch_size is 512 and using arch4 architecture, how can I adjust the parameters, Can you give Any suggestions? When num_epochs increased to 200, the corr value recovered at about 120 times, but then increased to the end.