Why is training so slow and GPU utilization so low?

qfgaohao / pytorch-ssd

MobileNetV1, MobileNetV2, VGG based SSD/SSD-lite implementation in Pytorch 1.0 / Pytorch 0.4. Out-of-box support for retraining on Open Images dataset. ONNX and Caffe2 support. Experiment Ideas like CoordConv.

https://medium.com/@smallfishbigsea/understand-ssd-and-implement-your-own-caa3232cd6ad

MIT License

1.39k stars 529 forks source link

Why is training so slow and GPU utilization so low? #150

Open Evie-0808 opened 3 years ago

Evie-0808 commented 3 years ago

When I use this code to train, why is the training speed so slow and the GPU utilization so low?

Nannigalaxy commented 3 years ago

Solved yet? Even I'm facing the same issue. Transformers may not be optimized properly?

Sanjay1911 commented 3 weeks ago

Did you figure out a way to solve this? I'm currently training a custom mbnet-ssd and my training is super slow in L4 GPU and there is no utilization of GPU.

I've also set use-cuda to true and my log says Using CUDA