Closed KojimaIsBad closed 2 years ago
Hi, KojimalsBad
yes. I don't recall precisely, however, it should take no more than an hour If your RAM supports, the best would be to move the data to your RAM to avoid reading from a HDD. One question is, did you not have other issues when training other models? I think this issue would also happen in other cases as well.
Thanks. I copied VoxCeleb1 to RAM and trained RawNet2 on it, then the training time dropped a lot. However, I cannot move the entire VoxCeleb2 to my RAM, maybe I need a new SSD.
I tried to train RawNet2 on VoxCeleb2 dataset with default settings. But I found that one epoch on average takes about 2.5 hours. By observing GPU activity, I found that most of the time GPU is waiting for data IO. To reduce GPU waiting time, I tried to increase args "num_workers" and "prefetch_factor" in PyTorch DataLoader,but the data loading time just did not decrease. My hardware: 3*RTX3090,128G Memory,HDD disk.