search

TaoRuijie / ECAPA-TDNN

Unofficial reimplementation of ECAPA-TDNN for speaker recognition (EER=0.86 for Vox1_O when train only in Vox2)
MIT License
594 stars 113 forks source link

How to speed up training? #29

Closed Drawlone closed 2 years ago

Drawlone commented 2 years ago

My GPU is also RTX 3090. Why is your training speed so fast? Mine is about 6 times slower!

TaoRuijie commented 2 years ago

use nvidia-smi to check the status of your GPU to make sure your GPU is running in the full speed. Which means GPU-Util is almost 100%

underdogliu commented 2 years ago

Yes I have got same problem on the runner. It roughly takes 2~2.5 seconds per-loop for 80-mel filterbank input. I'd like to ask if there is distributed setups or @TaoRuijie you used one TITAN X GPU from the beginning to the end. Of course I know there are some other specifications but this is the first information I'd like to know your kindly answer. Thanks.

TaoRuijie commented 2 years ago

No distributed steps, I only use one 3090 GPU from the beginning to the end.

Drawlone commented 2 years ago

看了您在blibili的为什么训练时GPU使用率低，跑的这么慢？磕盐经验分享，我已经找出了问题所在，感谢分享，受益良多！