Why the training is very slow?

[MuhammadAsadJaved]

Hi, I am training YOLOV3 with following details but training is very slow, about 14000 iterations after 8 days and nights continues training. GPU = TITAN V Model = YOLOV3 Classes = 6 Training set = 26935 visual images Solver.prototxt configurations are shown in the picture

What is the possible reason? The same model, on the same dataset can be trained for 100000 iterations in 5 days on darknet.

[eric612]

The possible reason is the image scale down cost too much time , e.x :when do image data augmentation and scaling down , 1920x1920 to 352x352

You can try to remove expand ratio , if it is used , I recommend to use lower resolution when produce lmdb

Asad notifications@github.com 於 2020年5月8日 週五 下午4:40 寫道：

Hi, I am training YOLOV3 with following details but training is very slow, about 14000 iterations after 8 days and nights continues training. GPU = TITAN V Model = YOLOV3 Classes = 6 Training set = 26935 visual images Solver.prototxt configurations are shown in the picture

What is the possible reason? The same model, on the same dataset can be trained for 100000 iterations in 5 days on darknet. [image: trainingSlow0-05-08 16-37-20] https://user-images.githubusercontent.com/28862708/81388032-433e8680-914a-11ea-9fff-b8b72a7f6254.png

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/eric612/MobileNet-YOLO/issues/253, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADYP2QELNPZO6EOJ6O4YMETRQPAN7ANCNFSM4M3766EQ .

[MuhammadAsadJaved]

Ok. I’ll give it a try . Thank you for your response.

On Fri, May 8, 2020 at 5:04 PM Eric Liu notifications@github.com wrote:

The possible reason is the image scale down cost too much time , e.x :when do image data augmentation and scaling down , 1920x1920 to 352x352

You can try to remove expand ratio , if it is used , I recommend to use lower resolution when produce lmdb

Asad notifications@github.com 於 2020年5月8日 週五 下午4:40 寫道：

Hi, I am training YOLOV3 with following details but training is very slow, about 14000 iterations after 8 days and nights continues training. GPU = TITAN V Model = YOLOV3 Classes = 6 Training set = 26935 visual images Solver.prototxt configurations are shown in the picture

What is the possible reason? The same model, on the same dataset can be trained for 100000 iterations in 5 days on darknet. [image: trainingSlow0-05-08 16-37-20] < https://user-images.githubusercontent.com/28862708/81388032-433e8680-914a-11ea-9fff-b8b72a7f6254.png

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/eric612/MobileNet-YOLO/issues/253, or unsubscribe < https://github.com/notifications/unsubscribe-auth/ADYP2QELNPZO6EOJ6O4YMETRQPAN7ANCNFSM4M3766EQ

.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/eric612/MobileNet-YOLO/issues/253#issuecomment-625718043, or unsubscribe https://github.com/notifications/unsubscribe-auth/AG4GR5BR6DZX2AAM2MWJMK3RQPDI7ANCNFSM4M3766EQ .