Open wenxie18 opened 4 years ago
wenxie18
commented
4 years ago Is there anyone else facing this problem? I guess my dataset has bad samples. I trained YOLO3 before with one dataset. The result is pretty good. But this time, when I train it with another dataset, there's always some nan/inf loss.
PieroCV
commented
4 years ago Early Sttoping callback and LROnPlateau use val_loss as a reference. If val_loss is a a not numeric value, these callbacks won't work. Now, the messages showed on console mean that the callbacks are not working properly because of nan values.
The solution of this problem is usually de .names file, if you pass a wrong .names file, the Dataset object will be corrupted and the train will return nan values. Another solution is to check tfrecord files. VoTT use to export some corrupted files.
wenxie18
commented
4 years ago Thank you for the suggestion, dear Piero. In fact, the inf/nan train loss appears within one epoch, which means it doesn't reach the validation process. Am I right?
Best, Wen
Piero Casusol notifications@github.com 于2020年3月4日周三 下午1:14写道：
Early Sttoping callback and LROnPlateau use val_loss as a reference. If val_loss is a a not numeric value, these callbacks won't work. Now, the messages showed on console mean that the callbacks are not working properly because of nan values.
The solution of this problem is usually de .names file, if you pass a wrong .names file, the Dataset object will be corrupted and the train will return nan values. Another solution is to check tfrecord files. VoTT use to export some corrupted files.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/zzh8829/yolov3-tf2/issues/186?email_source=notifications&email_token=AIGXRW7J2JGF7Y3FGJBP7CLRF2SCVA5CNFSM4KZVRIF2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENZWMRQ#issuecomment-594765382, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIGXRWZCNYEPYTXT5RUJLBLRF2SCVANCNFSM4KZVRIFQ .
-- Wen Xie, Ph.D. Student Department of Electrical and Computer Engineering W319, Engineering Building 2 4800 Calhoun Road University of Houston Houston, TX 77004-4005 Tele: (346)-256-6096 wxie5@uh.edu wxie5@uh.edu
B.ENG. Electronic Information Engineering, UESTCB.A. Finance, UESTC
2020-02-22 17:19:04.316272: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:971] 0 2020-02-22 17:19:04.318202: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:984] 0: N 2020-02-22 17:19:04.320169: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1097] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6392 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0, compute capability: 6.1) Create YOLOv3 model with 9 anchors and 5 classes. Load weights C:\Users\Dawei Chen\Desktop\Vince\hotel\m41\trained_weights_final.h5. Freeze the first 185 layers of total 252 layers. Train on 2200 samples, val on 549 samples, with batch size 8. Epoch 1/100 275/275 [==============================] - 104s 378ms/step - loss: 24.3365 - val_loss: 22.9203 Epoch 2/100 275/275 [==============================] - 93s 339ms/step - loss: nan - val_loss: nan C:\Users\Dawei Chen.conda\envs\vince\lib\site-packages\keras\callbacks.py:1090: RuntimeWarning: invalid value encountered in less self.monitor_op = lambda a, b: np.less(a, b - self.min_delta) C:\Users\Dawei Chen.conda\envs\vince\lib\site-packages\keras\callbacks.py:543: RuntimeWarning: invalid value encountered in less if self.monitor_op(current - self.min_delta, self.best): Epoch 3/100 135/275 [=============>................] - ETA: 37s - loss: nan (vince) C:\Users\Dawei Chen\Desktop\Vince\hotel\keras-yolo3-master>