search

Zhongdao / Towards-Realtime-MOT

Joint Detection and Embedding for fast multi-object tracking
MIT License
2.37k stars 539 forks source link

Training loss suddenly became nan on my custom dataset. What is the problem? #216

Open AlekseySh opened 3 years ago

AlekseySh commented 3 years ago 
ssh://osai@31.131.159.19:33342/home/osai/miniconda3/bin/python3.8 -u /home/osai/code/osai-players/osai_players/Towards-Realtime-MOT/train.py --data-cfg cfg/ccmcpe_example.json --cfg cfg/yolov3_1088x608.cfg --batch-size 2
================================================================================
dataset summary
OrderedDict([('example', 12.0)])
total # identities: 13
start index
OrderedDict([('example', 0)])
================================================================================
2021-02-15 13:13:57 [INFO]:    Epoch       Batch       box      conf        id     total  nTargets      time
2021-02-15 13:14:00 [INFO]:    0/199        0/29     0.187      2.05      5.16      85.7        21      2.79
2021-02-15 13:14:05 [INFO]:    Epoch       Batch       box      conf        id     total  nTargets      time
2021-02-15 13:14:06 [INFO]:    1/199        0/29    0.0309     0.013       3.9     -7.25        22     0.815
2021-02-15 13:14:12 [INFO]:    Epoch       Batch       box      conf        id     total  nTargets      time
2021-02-15 13:14:13 [INFO]:    2/199        0/29    0.0289    0.0502      5.08     -4.66        22      1.12
2021-02-15 13:14:19 [INFO]:    Epoch       Batch       box      conf        id     total  nTargets      time
2021-02-15 13:14:20 [INFO]:    3/199        0/29    0.0311    0.0663      4.75     -8.86        22      1.08
2021-02-15 13:14:26 [INFO]:    Epoch       Batch       box      conf        id     total  nTargets      time
2021-02-15 13:14:27 [INFO]:    4/199        0/29    0.0305    0.0239      3.69     -12.8        20      1.12
2021-02-15 13:14:33 [INFO]:    Epoch       Batch       box      conf        id     total  nTargets      time
2021-02-15 13:14:33 [INFO]:    5/199        0/29    0.0232    0.0217      4.48     -16.1        24      1.05
2021-02-15 13:14:40 [INFO]:    Epoch       Batch       box      conf        id     total  nTargets      time
2021-02-15 13:14:40 [INFO]:    6/199        0/29    0.0327    0.0185      4.22     -18.5        22      1.03
2021-02-15 13:14:46 [INFO]:    Epoch       Batch       box      conf        id     total  nTargets      time
2021-02-15 13:14:47 [INFO]:    7/199        0/29    0.0257    0.0196      4.41     -20.9        22      1.11
2021-02-15 13:14:53 [INFO]:    Epoch       Batch       box      conf        id     total  nTargets      time
2021-02-15 13:14:54 [INFO]:    8/199        0/29    0.0226    0.0282      4.73     -22.8        23      1.09
2021-02-15 13:15:00 [INFO]:    Epoch       Batch       box      conf        id     total  nTargets      time
2021-02-15 13:15:01 [INFO]:    9/199        0/29     0.025    0.0229       4.3       -25        19      1.13
2021-02-15 13:15:07 [INFO]:    Epoch       Batch       box      conf        id     total  nTargets      time
2021-02-15 13:15:08 [INFO]:   10/199        0/29    0.0288    0.0204      4.15     -26.9        22      1.03
2021-02-15 13:15:14 [INFO]:    Epoch       Batch       box      conf        id     total  nTargets      time
2021-02-15 13:15:15 [INFO]:   11/199        0/29    0.0233    0.0246      4.74     -28.5        23      1.18
2021-02-15 13:15:21 [INFO]:    Epoch       Batch       box      conf        id     total  nTargets      time
2021-02-15 13:15:21 [INFO]:   12/199        0/29    0.0264    0.0193      4.55     -30.7        20       1.1
2021-02-15 13:15:27 [INFO]:    Epoch       Batch       box      conf        id     total  nTargets      time
2021-02-15 13:15:28 [INFO]:   13/199        0/29    0.0329    0.0299      4.81     -32.3        22      1.01
2021-02-15 13:15:34 [INFO]:    Epoch       Batch       box      conf        id     total  nTargets      time
2021-02-15 13:15:35 [INFO]:   14/199        0/29    0.0229    0.0211      4.15     -34.8        22      1.01
2021-02-15 13:15:41 [INFO]:    Epoch       Batch       box      conf        id     total  nTargets      time
2021-02-15 13:15:42 [INFO]:   15/199        0/29    0.0303    0.0142       4.2     -36.7        22      1.09
2021-02-15 13:15:48 [INFO]:    Epoch       Batch       box      conf        id     total  nTargets      time
2021-02-15 13:15:48 [INFO]:   16/199        0/29    0.0203    0.0265      4.77     -38.7        22      1.04
2021-02-15 13:15:55 [INFO]:    Epoch       Batch       box      conf        id     total  nTargets      time
2021-02-15 13:15:55 [INFO]:   17/199        0/29    0.0208    0.0171      3.94       -41        21      1.12
2021-02-15 13:16:02 [INFO]:    Epoch       Batch       box      conf        id     total  nTargets      time
2021-02-15 13:16:02 [INFO]:   18/199        0/29    0.0169   0.00956         4     -43.2        21      1.06
2021-02-15 13:16:08 [INFO]:    Epoch       Batch       box      conf        id     total  nTargets      time
2021-02-15 13:16:09 [INFO]:   19/199        0/29    0.0224    0.0128       3.9     -44.9        21      1.12
2021-02-15 13:16:15 [INFO]:    Epoch       Batch       box      conf        id     total  nTargets      time
2021-02-15 13:16:16 [INFO]:   20/199        0/29    0.0184    0.0127      3.31     -47.1        22      1.14
2021-02-15 13:16:22 [INFO]:    Epoch       Batch       box      conf        id     total  nTargets      time
2021-02-15 13:16:23 [INFO]:   21/199        0/29    0.0348  5.01e+08      4.44  7.22e+08        22       1.2
2021-02-15 13:16:29 [INFO]:    Epoch       Batch       box      conf        id     total  nTargets      time
2021-02-15 13:16:30 [INFO]:   22/199        0/29    0.0243  4.68e+08      4.42   8.6e+08        21      1.11
afonseca18 commented 3 years ago

Hello! Have you solved this? I'm having the same issue. Thanks

256387110 commented 1 year ago

你好！ 你解决了这个问题吗？我遇到了同样的问题。 谢谢