search

erkil1452 / gaze360

Code for the Gaze360: Physically Unconstrained Gaze Estimation in the Wild Dataset
http://gaze360.csail.mit.edu
Other
225 stars 42 forks source link

some error in the traindata #43

Closed jxncyym closed 2 years ago

jxncyym commented 2 years ago

@erkil1452 I trained a model using your train.txt, but there are some warning, I guess there are some error in you training data:

Epoch: [325][651/1587] Time 0.721 (0.724) Data 0.010 (0.013) Angular 2.076 (2.421) Loss 0.0120 (0.0128) Prediction Error 2.5859 (2.5833) Epoch: [325][652/1587] Time 0.722 (0.724) Data 0.009 (0.013) Angular 2.126 (2.421) Loss 0.0104 (0.0128) Prediction Error 2.4450 (2.5831) Epoch: [325][653/1587] Time 0.721 (0.724) Data 0.010 (0.013) Angular 2.402 (2.421) Loss 0.0133 (0.0128) Prediction Error 2.4985 (2.5830) WARNING:root:NaN or Inf found in input tensor. Epoch: [325][654/1587] Time 0.721 (0.724) Data 0.010 (0.013) Angular nan (nan) Loss 0.0152 (0.0128) Prediction Error 2.5007 (2.5829) Epoch: [325][655/1587] Time 0.721 (0.724) Data 0.010 (0.013) Angular 2.482 (nan) Loss 0.0134 (0.0128) Prediction Error 2.5017 (2.5828) Epoch: [325][656/1587] Time 0.720 (0.724) Data 0.010 (0.012) Angular 2.701 (nan) Loss 0.0138 (0.0128) Prediction Error 2.8752 (2.5832) Epoch: [325][657/1587] Time 0.721 (0.724) Data 0.010 (0.012) Angular 1.995 (nan) Loss 0.0106 (0.0128) Prediction Error 2.3353 (2.5828) Epoch: [325][658/1587] Time 0.723 (0.724) Data 0.010 (0.012) Angular 2.179 (nan) Loss 0.0112 (0.0128) Prediction Error 2.4611 (2.5826) Epoch: [325][659/1587] Time 0.723 (0.724) Data 0.010 (0.012) Angular 2.341 (nan) Loss 0.0140 (0.0128) Prediction Error 2.5880 (2.5826) Epoch: [325][660/1587] Time 0.721 (0.724) Data 0.010 (0.012) Angular 2.273 (nan) Loss 0.0114 (0.0128) Prediction Error 2.2808 (2.5822) Epoch: [325][661/1587] Time 0.720 (0.724) Data 0.010

erkil1452 commented 2 years ago

It seems you are getting NaN. You may need to investigate further into your model, data loader, training parameters etc. It is not easy to find culprit in these cases. I would try plotting the gradients over the course of the training.

jxncyym commented 2 years ago

@erkil1452 I use the data in your train.txt, and the model predict the gaze using only single RGB image. so my training data is only within the train.txt, each line in the train.txt is a training data.

erkil1452 commented 2 years ago

I am noticing you get NaN only for the angular error. If this is the source of your issues I suspect that you have not clamped the input values you pass into a acos or asin to range [-1,1]. Just a possibility.

jxncyym commented 2 years ago

@erkil1452

  1. I change something like the model and the dataset loader modules based on your code. the dataset loader only change the input of 7 images to only one image， other not changed. the nan value not appeared in every epoch, maybe is a normal value in the next epoch.

  2. what the effect will be caused by the nan value,the loss will not be decrease then the model will not be the best?

  3. now my training log is: Epoch: [393][26/1587] Time 0.721 (0.795) Data 0.010 (0.084) Angular 2.163 (2.203) Loss 0.0114 (0.0115) Prediction Error 2.3297 (2.4135) Epoch: [393][27/1587] Time 0.721 (0.792) Data 0.010 (0.081) Angular 2.161 (2.201) Loss 0.0108 (0.0115) Prediction Error 2.2852 (2.4089) Epoch: [393][28/1587] Time 0.721 (0.790) Data 0.009 (0.079) Angular 1.797 (2.187) Loss 0.0093 (0.0114) Prediction Error 2.1193 (2.3990) Epoch: [393][29/1587] Time 0.721 (0.787) Data 0.010 (0.077) Angular 2.118 (2.185) Loss 0.0114 (0.0114) Prediction Error 2.2670 (2.3946) Epoch: [393][30/1587] Time 0.722 (0.785) Data 0.010 (0.074) Angular 1.970 (2.178) Loss 0.0101 (0.0114) Prediction Error 2.1648 (2.3871) Epoch: [393][31/1587] Time 0.722 (0.783) Data 0.010 (0.072) Angular 2.221 (2.179) Loss 0.0114 (0.0114) Prediction Error 2.2715 (2.3835) Epoch: [393][32/1587] Time 0.722 (0.781) Data 0.010 (0.071) Angular 2.109 (2.177) Loss 0.0108 (0.0114) Prediction Error 2.2226 (2.3787) Epoch: [393][33/1587] Time 0.722 (0.780) Data 0.010 (0.069) Angular 2.022 (2.173) Loss 0.0107 (0.0114) Prediction Error 2.3832 (2.3788)

the validate log is: Epoch: [178/213] Time 0.218 (0.234) Angular 13.9332 (16.3664) Loss 0.1301 (0.1761) Epoch: [179/213] Time 0.217 (0.234) Angular 18.2591 (16.3769) Loss 0.1606 (0.1760) Epoch: [180/213] Time 0.218 (0.233) Angular 15.4334 (16.3717) Loss 0.2238 (0.1762) Epoch: [181/213] Time 0.219 (0.233) Angular 17.6182 (16.3786) Loss 0.1944 (0.1763) Epoch: [182/213] Time 0.218 (0.233) Angular 17.1792 (16.3830) Loss 0.2205 (0.1766) Epoch: [183/213] Time 0.219 (0.233) Angular 18.0038 (16.3918) Loss 0.2387 (0.1769) Epoch: [184/213] Time 0.219 (0.233) Angular 15.4107 (16.3865) Loss 0.1948 (0.1770) Epoch: [185/213] Time 0.220 (0.233) Angular 15.2042 (16.3801) Loss 0.1492 (0.1769) Epoch: [186/213] Time 0.218 (0.233) Angular 19.2744 (16.3956) Loss 0.2371 (0.1772)

the model in that status is the best? the validate angular error is still large. Is the model overfitting? what suggestion do you give when the model overfitting?

erkil1452 commented 2 years ago

The NaN would completely spoil the network weights and make the model unusable.

If your model is overfitting you can either try to enlarge your training set, add augmentation to your training data, add noise, Dropouts, ... or just stop the training early.