search

dbolya / yolact

A simple, fully convolutional model for real-time instance segmentation.
MIT License
5k stars 1.33k forks source link

why trainning is slow #245

Open kakaluote opened 4 years ago

kakaluote commented 4 years ago

my environment is: ubuntu 18.04, cpu i5-7500, GTX1070, cuda10.0, pytorch1.3.1

[  0]      70 || B: 4.143 | C: 4.777 | M: 5.276 | S: 0.683 | T: 14.878 || ETA: 19 days, 7:47:49 || timer: 0.349
other time 0.0009167194366455078
net time 0.3522038459777832
[  0]      71 || B: 4.143 | C: 4.749 | M: 5.288 | S: 0.674 | T: 14.853 || ETA: 19 days, 5:42:55 || timer: 0.353
other time 0.0005395412445068359
net time 0.3603506088256836
[  0]      72 || B: 4.132 | C: 4.715 | M: 5.260 | S: 0.665 | T: 14.772 || ETA: 19 days, 3:47:19 || timer: 0.361
other time 0.0007359981536865234
net time 0.3483552932739258
[  0]      73 || B: 4.145 | C: 4.724 | M: 5.254 | S: 0.664 | T: 14.787 || ETA: 19 days, 1:46:12 || timer: 0.349
other time 0.0006895065307617188
net time 0.34925293922424316
[  0]      74 || B: 4.124 | C: 4.691 | M: 5.236 | S: 0.656 | T: 14.707 || ETA: 18 days, 23:48:54 || timer: 0.350
other time 0.0007295608520507812
net time 0.3484385013580322
[  0]      75 || B: 4.129 | C: 4.674 | M: 5.230 | S: 0.651 | T: 14.685 || ETA: 18 days, 21:54:10 || timer: 0.349
other time 0.0005517005920410156
net time 0.34479689598083496
[  0]      76 || B: 4.131 | C: 4.650 | M: 5.244 | S: 0.645 | T: 14.670 || ETA: 18 days, 19:59:42 || timer: 0.345
other time 0.0006573200225830078
net time 0.35280370712280273
[  0]      77 || B: 4.139 | C: 4.632 | M: 5.242 | S: 0.640 | T: 14.653 || ETA: 18 days, 18:13:43 || timer: 0.354
other time 0.0007016658782958984
net time 0.3458712100982666
[  0]      78 || B: 4.146 | C: 4.614 | M: 5.251 | S: 0.634 | T: 14.645 || ETA: 18 days, 16:25:49 || timer: 0.347
other time 0.0012123584747314453
net time 0.3539700508117676
[  0]      79 || B: 4.137 | C: 4.597 | M: 5.244 | S: 0.631 | T: 14.609 || ETA: 18 days, 14:46:18 || timer: 0.355
other time 0.000522613525390625
net time 0.3708500862121582
[  0]      80 || B: 4.126 | C: 4.576 | M: 5.242 | S: 0.626 | T: 14.570 || ETA: 18 days, 13:19:56 || timer: 0.371
other time 0.0004887580871582031
net time 0.3785238265991211
[  0]      81 || B: 4.114 | C: 4.559 | M: 5.236 | S: 0.624 | T: 14.532 || ETA: 18 days, 12:00:36 || timer: 0.379
other time 0.14234662055969238
net time 0.3470127582550049
[  0]      82 || B: 4.113 | C: 4.538 | M: 5.252 | S: 0.618 | T: 14.522 || ETA: 18 days, 11:54:06 || timer: 0.489
other time 0.001743316650390625
net time 0.37055015563964844
[  0]      83 || B: 4.110 | C: 4.519 | M: 5.235 | S: 0.613 | T: 14.476 || ETA: 18 days, 10:33:30 || timer: 0.372
other time 0.001003265380859375
net time 0.36176180839538574
[  0]      84 || B: 4.108 | C: 4.496 | M: 5.209 | S: 0.607 | T: 14.420 || ETA: 18 days, 9:08:48 || timer: 0.363
other time 0.0005767345428466797
net time 0.3541433811187744
[  0]      85 || B: 4.102 | C: 4.471 | M: 5.214 | S: 0.601 | T: 14.388 || ETA: 18 days, 7:41:03 || timer: 0.355
other time 0.00058746337890625
net time 0.36074399948120117
[  0]      86 || B: 4.110 | C: 4.444 | M: 5.188 | S: 0.596 | T: 14.339 || ETA: 18 days, 6:19:23 || timer: 0.361
other time 0.0007290840148925781
net time 0.39004969596862793
[  0]      87 || B: 4.104 | C: 4.421 | M: 5.190 | S: 0.590 | T: 14.304 || ETA: 18 days, 5:17:25 || timer: 0.391
other time 0.303236722946167
net time 0.36629438400268555
[  0]      88 || B: 4.090 | C: 4.399 | M: 5.176 | S: 0.585 | T: 14.249 || ETA: 18 days, 7:03:51 || timer: 0.670
other time 0.0005018711090087891
net time 0.385819673538208
[  0]      89 || B: 4.075 | C: 4.381 | M: 5.149 | S: 0.580 | T: 14.186 || ETA: 18 days, 6:00:05 || timer: 0.386
other time 0.0006783008575439453
net time 0.36234092712402344
[  0]      90 || B: 4.068 | C: 4.360 | M: 5.131 | S: 0.575 | T: 14.135 || ETA: 18 days, 4:44:04 || timer: 0.363
other time 0.0005276203155517578
net time 0.37754034996032715
[  0]      91 || B: 4.071 | C: 4.340 | M: 5.132 | S: 0.569 | T: 14.113 || ETA: 18 days, 3:38:27 || timer: 0.378
other time 0.000518798828125
net time 0.37877702713012695
[  0]      92 || B: 4.062 | C: 4.315 | M: 5.113 | S: 0.563 | T: 14.053 || ETA: 18 days, 2:34:56 || timer: 0.379
other time 0.0004725456237792969
net time 0.36324310302734375
[  0]      93 || B: 4.041 | C: 4.294 | M: 5.097 | S: 0.558 | T: 13.990 || ETA: 18 days, 1:23:54 || timer: 0.364
other time 0.0005574226379394531
net time 0.3681919574737549
[  0]      94 || B: 4.037 | C: 4.275 | M: 5.101 | S: 0.553 | T: 13.966 || ETA: 18 days, 0:17:14 || timer: 0.369
other time 0.0033638477325439453
net time 0.3677830696105957
[  0]      95 || B: 4.022 | C: 4.253 | M: 5.069 | S: 0.549 | T: 13.893 || ETA: 17 days, 23:13:19 || timer: 0.371
other time 0.671623945236206
net time 0.3598153591156006
[  0]      96 || B: 4.014 | C: 4.237 | M: 5.082 | S: 0.544 | T: 13.877 || ETA: 18 days, 4:13:44 || timer: 1.032
other time 0.0006768703460693359
net time 0.3947114944458008
[  0]      97 || B: 4.007 | C: 4.220 | M: 5.092 | S: 0.539 | T: 13.858 || ETA: 18 days, 3:21:51 || timer: 0.395
other time 0.004936933517456055
net time 0.3583662509918213
[  0]      98 || B: 3.998 | C: 4.204 | M: 5.090 | S: 0.534 | T: 13.825 || ETA: 18 days, 2:16:03 || timer: 0.368
other time 0.005922555923461914
net time 0.3604300022125244
[  0]      99 || B: 3.981 | C: 4.182 | M: 5.071 | S: 0.529 | T: 13.764 || ETA: 18 days, 1:10:56 || timer: 0.366
other time 0.07806825637817383
net time 0.3562493324279785

i've added some code to print net forward time, the batch size is 2, net forward time is 350ms. the forward time is around 50ms when eval model, why trainning forward is so slow

kakaluote commented 4 years ago

i add more prints, it shows: net forward use 50ms calc loss use 60ms backward use 50ms optimizer use 200ms !!!!

dbolya commented 4 years ago

It takes ~6 days on a GTX 1080ti, but a 1080ti should not be 3x as fast as a 1070. What batch size are you using? Maybe it's not fitting into your GPU and it's paging to memory every iteration or something.