Closed zzxiang closed 1 year ago
Thank you for your detailed report.
This crash may be caused when all of samples in batch are shorter than batch_max_steps
.
And when using multi-gpu, each gpu's batch size will be batch size / #gpus, i.e., the acutual batch size in each gpu will be decreased.
Therefore, the issue more likely to occur.
So in conclusion, this is expected behavior.
To solve this problem, please use remove_short_samples=true
or reducing batch_max_steps
.
Thanks for your quick reply!
I'll try reducing batch_max_steps
since the data is already small and I want to keep all the data as possible.
But I am still a little curious about two more questions.
Does reducing batch_max_steps
cause a bad effect on training accuracy?
The crash actually happened after 326 epoch training has finished. But if a gpu batch size being shorter than batch_max_steps
will definitely cause a crash, why doesn't the crash happen at the first epoch?
Does reducing batch_max_steps cause a bad effect on training accuracy?
In my experiences, the effect is limited (I tried halved or double). But too short may not be suitable.
But if a gpu batch size being shorter than batch_max_steps will definitely cause a crash, why doesn't the crash happen at the first epoch?
This is because in each epoch the data is shuffled and create new batches. The crash will be only happened when unfortunately a batch contains only short samples.
Thank you very much! Very easy to understand.
Crash Description
The following exception is thrown in
train.log
. Seems that the training program crashed because the wav length of training data is too short. (I setremove_short_samples
tofalse
inparallel_wavegan.v1.yaml
.)But the same config and data doesn't crash if
n_gpus
is set to 1 inrun.sh
, so it may have something to do with distributed computing.Training Description
The data contains 200 wav files, with 198 files set as training set, and the remaining 2 files set as both dev and eval.
n_gpus
is set to 2 since I've got 2 GPUs.Here's the relating parameters in
run.sh
I used the
parallel_wavegan.v1.yaml
which comes from the pretrained JSUT pwg model, withremove_short_samples
set tofalse
.Environment
ParrallelWaveGAN: v0.5.5 Python: 3.8.10
The output of
pip list
:The output of
nvidia-smi
The output of
nvcc -V
: