search

DeLightCMU / CASD

85 stars 21 forks source link

About the training speed #7

Open 1170300714 opened 3 years ago

1170300714 commented 3 years ago

Thanks for your great job! I train your model on TITAN RTX and the speed is about 2.7s/iter. Is the speed the same as yours? Or maybe it’s a problem with my GPU. Looking forward to your reply.

AlphaGoMK commented 3 years ago

Hi, I've trained vgg16-VOC model on Tesla M40 GPUs, and my speed is about 6.8s/iter. The training speed is too long so I did not finish it.
And I noticed GPU-Mem consumption is imbalanced among multi-gpus (varys from 6G to 13G)
Is this phenomenon same as yours?

Justinhzy commented 3 years ago

Hi, My training speed is 1.9s/iter without parallel. I am still preparing the paralleled version, it should be faster.

1170300714 commented 3 years ago

Hi, My training speed is 1.9s/iter without parallel. I am still preparing the paralleled version, it should be faster.

Thanks a lot

1170300714 commented 3 years ago

Hi, I've trained vgg16-VOC model on Tesla M40 GPUs, and my speed is about 6.8s/iter. The training speed is too long so I did not finish it. And I noticed GPU-Mem consumption is imbalanced among multi-gpus (varys from 6G to 13G) Is this phenomenon same as yours?

Yes, I met the phenomenon too.