asusdisciple
commented
8 months ago I was able to solve the issue. The problem lies with the train_loader class. Most of the "training time" results from the heavy dataloading in a multi gpu setting. By default the workers are dismissed after every epoch, so the dataset needs to be loaded again. By using persistent_workers=True the training time can be reduced to 13 sec/Epoch on 4 GPUs. Its still not optimal since actual training is only performed during a few seconds of this time frame, but it is still a large improvement.
Another thing I noticed was to set the batch_size down to a rather small size, since the loading to the device in
for i, batch in pb:
if rank == 0:
start_b = time.time()
x, y, _, y_mel = batch
x = x.to(device, non_blocking=True)
y = y.to(device, non_blocking=True)
y_mel = y_mel.to(device, non_blocking=True)
y = y.unsqueeze(1)took a long time. Now one epoch was trained in 7 seconds. Just wanted to let you guys know, in case you run into these problems.
HninLwin-byte
RF5
First of all thanks for the great model! I tested it extensively by now and ran across a few problems and performance issues which you might can help with.
Matching takes a lot of time with big datasets (1000+ 2min files), since it is not multi-gpu, do you intend to change that in the future?
General behaviour: For training in general, it seems to be better to have a few big files rather than many small files (2min vs 10sec). I think this might be related to the overhead introduced by all the small .pt "models". Can you confirm this or believe this is plausible?
My biggest issue so far is when I try to fine tune the hifi-gan vocoder. My Notebook with a A4500 seems to be on par, even outperforming my DGX Station with 4 x V100 32GB GPUs, which is strange.
I identified the following things:
During validation the operation is performed on all files rather than in a batch (1000+ files). The station and the notebook are both about equally fast in validating all files. However the station uses 4 GPUs which are working at 100% all the time and should be a lot faster. Since this is really slow how often do you think should I perform validation?
During batch training the notebook also outperforms the station by a little bit completing one epoch in about 40sec (station 48sec). However when I look at nvidia-smi on the station the GPU usage is at 0% all the time.
Unfortunately there seem to be some serious issues with a multi-gpu approach. If I only use one GPU on the station one epoch takes about 17 seconds. Maybe you have an idea what goes wrong here?
Edit: When I monitored the epochs in a multi-gpu setting it seems the epoch itself was trained really fast in 5 seconds. However before the progressbar appears there seems to happen some loading which takes the other 50 seconds. Do you know which process is responsible for that time gap or how to minmize it?
What I tried so far: I tried to use different batch sizes and adjusted the number of workers in the config, but it did no really change results that much,