Closed joonson closed 3 years ago
@joonson Have added it to the code? I can't find it.Thanks!
Not yet, hence the open issue
Added in a new branch distributed
. This is the configuration used to produce EER 1.1771
in the released pre-trained model. Note that 8 GPUs were used to train this model, so test_interval
and max_epoch
must be changed accordingly if you want to use a different number of GPUs.
@joonson 8 different gpus or 8 cores on a single gpu?
Also in this comment there is separate config vs one mentioned above. Are both of them producing same results?
Thank you always. you said we must change the configuration when we use that code with different GPU environment. Could you recommend configuration for 2 GPU?
@zeek-han The number of distributed GPU affects the training in the same way changing the batch size does. You will need to experiment to find out what works best.
@009deep Very similar, but not exactly the same.
Added in a new branch
distributed
. This is the configuration used to produceEER 1.1771
in the released pre-trained model. Note that 8 GPUs were used to train this model, sotest_interval
andmax_epoch
must be changed accordingly if you want to use a different number of GPUs.
Hi, @joonson. I test This configuration but the EER is 2.4%. Note that it only trains for 36 epochs. Is that right?
@lawlict Did you train using distributed training with 8 GPUs?
@joonson Yes, of course. I think the number of training steps is too small.
Also in this comment there is separate config vs one mentioned above. Are both of them producing same results?
Another configuration here takes much more training steps, but I don't have time to test it now.
DistributedDataParallel
(single node for now)DataParallel
is no longer recommended