questions about model training

yuyq96 / D-TDNN

PyTorch implementation of Densely Connected Time Delay Neural Network

83 stars 24 forks source link

questions about model training #4

Open forwiat opened 3 years ago

forwiat commented 3 years ago

hello, yuyq96, Thank you so much for the great work you've shared. I learned that D-TDNNSS mini-batch setting 128 from D-TDNN paper. But this model is too large to train on single gpu. Could you tell me how you train it? Using nn.Parallel or DDP? Looking forward to you reply

yuyq96 commented 3 years ago

Actually, it should fit in a GPU with 12GB RAM when memory_efficient set to True, which is the default. If not, you can try:

Using nn.DataParallel to train the model on two cards, and the mini-batch size on each card is 64, which should also be large enough for the batch normalization.
Slightly decreasing the mini-batch size, e.g. to 100, then see if it fits in your card and how much RAM it consumes.

forwiat commented 3 years ago

Ok, I will try it. Thanks a lot

shgidi commented 1 year ago

@yuyq96 Hi, do you have a training script for this model?

yuyq96 commented 1 year ago

@shgidi The original training script for D-TDNN or CAM are not open source due to the lack of company procedure. However, now you can use 3D-Speaker to train D-TDNN, CAM and CAM++. This project is lead by my former colleagues at Alibaba DAMO, and it is similar to the original training script.