Closed sunnysd closed 5 years ago
Hi,
I have trained Fast-SCNN with coco dataset for ~20 epochs, the inference output has nothing(entire image is blank). Please can throw so tips to debug this issue. Do i need to do any preprocessing on coco dataset. Highly appreciate your help on this.
training logs: Epoch: [18/160] Iter [1694/1927] || Time: 143237.6596 sec || lr: 0.00893152 || Loss: 0.6481 Epoch: [18/160] Iter [1704/1927] || Time: 143277.4028 sec || lr: 0.00893123 || Loss: 0.6621 Epoch: [18/160] Iter [1714/1927] || Time: 143315.9710 sec || lr: 0.00893093 || Loss: 0.5438 Epoch: [18/160] Iter [1724/1927] || Time: 143354.9870 sec || lr: 0.00893064 || Loss: 0.7725 Epoch: [18/160] Iter [1734/1927] || Time: 143394.4705 sec || lr: 0.00893034 || Loss: 0.5357 Epoch: [18/160] Iter [1744/1927] || Time: 143434.3466 sec || lr: 0.00893005 || Loss: 0.5382 Epoch: [18/160] Iter [1754/1927] || Time: 143475.0332 sec || lr: 0.00892975 || Loss: 0.4904 Epoch: [18/160] Iter [1764/1927] || Time: 143515.1107 sec || lr: 0.00892945 || Loss: 0.5637 Epoch: [18/160] Iter [1774/1927] || Time: 143554.9711 sec || lr: 0.00892916 || Loss: 0.6457 Epoch: [18/160] Iter [1784/1927] || Time: 143594.5968 sec || lr: 0.00892886 || Loss: 0.4945 Epoch: [18/160] Iter [1794/1927] || Time: 143633.9663 sec || lr: 0.00892857 || Loss: 0.3928 Epoch: [18/160] Iter [1804/1927] || Time: 143674.1125 sec || lr: 0.00892827 || Loss: 0.4256 Epoch: [18/160] Iter [1814/1927] || Time: 143712.8005 sec || lr: 0.00892798 || Loss: 0.7373 Epoch: [18/160] Iter [1824/1927] || Time: 143752.1207 sec || lr: 0.00892768 || Loss: 0.4910 Epoch: [18/160] Iter [1834/1927] || Time: 143791.0121 sec || lr: 0.00892738 || Loss: 0.6223 Epoch: [18/160] Iter [1844/1927] || Time: 143831.8926 sec || lr: 0.00892709 || Loss: 0.6388 Epoch: [18/160] Iter [1854/1927] || Time: 143870.4438 sec || lr: 0.00892679 || Loss: 0.6908 Epoch: [18/160] Iter [1864/1927] || Time: 143909.4174 sec || lr: 0.00892650 || Loss: 0.4131 Epoch: [18/160] Iter [1874/1927] || Time: 143948.1374 sec || lr: 0.00892620 || Loss: 0.4725 Epoch: [18/160] Iter [1884/1927] || Time: 143987.2357 sec || lr: 0.00892591 || Loss: 0.6805 Epoch: [18/160] Iter [1894/1927] || Time: 144026.5409 sec || lr: 0.00892561 || Loss: 0.7085 Epoch: [18/160] Iter [1904/1927] || Time: 144066.6354 sec || lr: 0.00892532 || Loss: 0.6547 Epoch: [18/160] Iter [1914/1927] || Time: 144105.0635 sec || lr: 0.00892502 || Loss: 0.4116 Epoch: [18/160] Iter [1924/1927] || Time: 144145.9772 sec || lr: 0.00892472 || Loss: 0.5329 Epoch: [19/160] Iter [ 7/1927] || Time: 144185.6879 sec || lr: 0.00892443 || Loss: 0.5201 Epoch: [19/160] Iter [ 17/1927] || Time: 144224.2033 sec || lr: 0.00892413 || Loss: 0.6291 Epoch: [19/160] Iter [ 27/1927] || Time: 144263.8119 sec || lr: 0.00892384 || Loss: 0.8196 Epoch: [19/160] Iter [ 37/1927] || Time: 144304.1935 sec || lr: 0.00892354 || Loss: 0.6305 Epoch: [19/160] Iter [ 47/1927] || Time: 144342.7663 sec || lr: 0.00892325 || Loss: 0.4930 Epoch: [19/160] Iter [ 57/1927] || Time: 144382.1391 sec || lr: 0.00892295 || Loss: 0.4747 Epoch: [19/160] Iter [ 67/1927] || Time: 144420.8839 sec || lr: 0.00892265 || Loss: 0.5396 Epoch: [19/160] Iter [ 77/1927] || Time: 144460.3386 sec || lr: 0.00892236 || Loss: 0.5535 Epoch: [19/160] Iter [ 87/1927] || Time: 144499.7273 sec || lr: 0.00892206 || Loss: 0.5598 Epoch: [19/160] Iter [ 97/1927] || Time: 144539.2534 sec || lr: 0.00892177 || Loss: 0.4794
Hi,
I have trained Fast-SCNN with coco dataset for ~20 epochs, the inference output has nothing(entire image is blank). Please can throw so tips to debug this issue. Do i need to do any preprocessing on coco dataset. Highly appreciate your help on this.
training logs: Epoch: [18/160] Iter [1694/1927] || Time: 143237.6596 sec || lr: 0.00893152 || Loss: 0.6481 Epoch: [18/160] Iter [1704/1927] || Time: 143277.4028 sec || lr: 0.00893123 || Loss: 0.6621 Epoch: [18/160] Iter [1714/1927] || Time: 143315.9710 sec || lr: 0.00893093 || Loss: 0.5438 Epoch: [18/160] Iter [1724/1927] || Time: 143354.9870 sec || lr: 0.00893064 || Loss: 0.7725 Epoch: [18/160] Iter [1734/1927] || Time: 143394.4705 sec || lr: 0.00893034 || Loss: 0.5357 Epoch: [18/160] Iter [1744/1927] || Time: 143434.3466 sec || lr: 0.00893005 || Loss: 0.5382 Epoch: [18/160] Iter [1754/1927] || Time: 143475.0332 sec || lr: 0.00892975 || Loss: 0.4904 Epoch: [18/160] Iter [1764/1927] || Time: 143515.1107 sec || lr: 0.00892945 || Loss: 0.5637 Epoch: [18/160] Iter [1774/1927] || Time: 143554.9711 sec || lr: 0.00892916 || Loss: 0.6457 Epoch: [18/160] Iter [1784/1927] || Time: 143594.5968 sec || lr: 0.00892886 || Loss: 0.4945 Epoch: [18/160] Iter [1794/1927] || Time: 143633.9663 sec || lr: 0.00892857 || Loss: 0.3928 Epoch: [18/160] Iter [1804/1927] || Time: 143674.1125 sec || lr: 0.00892827 || Loss: 0.4256 Epoch: [18/160] Iter [1814/1927] || Time: 143712.8005 sec || lr: 0.00892798 || Loss: 0.7373 Epoch: [18/160] Iter [1824/1927] || Time: 143752.1207 sec || lr: 0.00892768 || Loss: 0.4910 Epoch: [18/160] Iter [1834/1927] || Time: 143791.0121 sec || lr: 0.00892738 || Loss: 0.6223 Epoch: [18/160] Iter [1844/1927] || Time: 143831.8926 sec || lr: 0.00892709 || Loss: 0.6388 Epoch: [18/160] Iter [1854/1927] || Time: 143870.4438 sec || lr: 0.00892679 || Loss: 0.6908 Epoch: [18/160] Iter [1864/1927] || Time: 143909.4174 sec || lr: 0.00892650 || Loss: 0.4131 Epoch: [18/160] Iter [1874/1927] || Time: 143948.1374 sec || lr: 0.00892620 || Loss: 0.4725 Epoch: [18/160] Iter [1884/1927] || Time: 143987.2357 sec || lr: 0.00892591 || Loss: 0.6805 Epoch: [18/160] Iter [1894/1927] || Time: 144026.5409 sec || lr: 0.00892561 || Loss: 0.7085 Epoch: [18/160] Iter [1904/1927] || Time: 144066.6354 sec || lr: 0.00892532 || Loss: 0.6547 Epoch: [18/160] Iter [1914/1927] || Time: 144105.0635 sec || lr: 0.00892502 || Loss: 0.4116 Epoch: [18/160] Iter [1924/1927] || Time: 144145.9772 sec || lr: 0.00892472 || Loss: 0.5329 Epoch: [19/160] Iter [ 7/1927] || Time: 144185.6879 sec || lr: 0.00892443 || Loss: 0.5201 Epoch: [19/160] Iter [ 17/1927] || Time: 144224.2033 sec || lr: 0.00892413 || Loss: 0.6291 Epoch: [19/160] Iter [ 27/1927] || Time: 144263.8119 sec || lr: 0.00892384 || Loss: 0.8196 Epoch: [19/160] Iter [ 37/1927] || Time: 144304.1935 sec || lr: 0.00892354 || Loss: 0.6305 Epoch: [19/160] Iter [ 47/1927] || Time: 144342.7663 sec || lr: 0.00892325 || Loss: 0.4930 Epoch: [19/160] Iter [ 57/1927] || Time: 144382.1391 sec || lr: 0.00892295 || Loss: 0.4747 Epoch: [19/160] Iter [ 67/1927] || Time: 144420.8839 sec || lr: 0.00892265 || Loss: 0.5396 Epoch: [19/160] Iter [ 77/1927] || Time: 144460.3386 sec || lr: 0.00892236 || Loss: 0.5535 Epoch: [19/160] Iter [ 87/1927] || Time: 144499.7273 sec || lr: 0.00892206 || Loss: 0.5598 Epoch: [19/160] Iter [ 97/1927] || Time: 144539.2534 sec || lr: 0.00892177 || Loss: 0.4794