Closed Cyp9715 closed 6 months ago
This is not intended.
Did you check CPU memory usage?
In the config, allow_cache: true
will store all data in CPU memory.
Therefore, if the CPU memory is full, the cache will be something problem (e.g., using too much swap instead cpu memory).
This may affect the training speed.
I think what you are concerned about is correct. It is currently operating normally. Resolved by significantly increasing the virtual memory swap size.
Thank you for your response.
From approximately 2000 epoch, it is learned at a fast rate of 5 epoch per second based on RTX3070, and then suddenly the speed drops sharply. I'm train parallel-waveGAN, and I've modified some of the datasets and conf files to replace them with other languages. Starting with 2000 epoch, I start reading SSD like crazy instead of GPU resources. I waited for more than 6 hours, but only five epoch progressed.
Is this intended? Thank you.
Below is the information about the setting file that I modified.
FILE : conf/parallel_wavegan.v3.yaml
FILE : train.log
Since then, I have been waiting for more than 6 hours, but only 5 epoch progressed.