search

tusen-ai / simpledet

A Simple and Versatile Framework for Object Detection and Instance Recognition
Apache License 2.0
3.08k stars 488 forks source link

Training speed is very slow #312

Open 1179021477 opened 4 years ago

1179021477 commented 4 years ago

@RogerChern I train tridentNet_1x with resnet50 on 4 GPU (a machine with 8 GPU), and I need 2 days. Especially, when others use other left GPUs in my machines, the speed of training my models is slower. Is there any way to make training faster? Like how to construct multi-thread, etc. My machine is TITAN X (Pascal).

xchani commented 4 years ago

Our dataloader does use multi-threading to load images. According to your description, you are sharing gpu server with others, then jobs from others may occupy cpu resource in that server, which slow down your training. Also the speed of disk(IOPS) is another major factor should be considered.