GPU memory

[ppl9115]

Hello Dear Author, Thank you for providing such clear and distinct code !

However I have some questions:

1. I want to know how much GPU memory you used for this training.
2. I encountered the problem of lacking GPU memory during training, so I tried to reduce the amount of dataset, but I am not sure where to place the dataset. Can I ask the author which place can be used to change ?

[yxlu-0102]

1. I typically use two GPUs with 24 GB of memory each for training.
2. If you want to change the amount of dataset, you just need to edit the number of data indexes in MP-SENet/VoiceBank+DEMAND/training.txt and test.txt.

[ppl9115]

Dear Author,

Thank you for responding to my question. I'm not sure if the GPU memory used for this training can be reduced ... Can I try to change the dense_channel to achieve this goal, or change some parameters (except batch_size)?

[yxlu-0102]

1. Sure, you can reduce GPU memory usage by decreasing the number of dense channels or by reducing the segment size in the config.json. However, these changes may affect the final training results.
2. By the way, you need to determine whether your GPU memory is insufficient during training or validation. During validation, the audio sequences are not truncated, which can cause variations in GPU memory usage. If you are experiencing memory issues only during validation, you can implement truncation as a solution.

[ppl9115]

Dear Author,

I try to decrease the number of dense channels in config.json to make it train successfully. However, when I implement the inference, I decreasing the number of dense channels in best_ckpt/config.json , it would appear size mismatch problem, I want to know the config.json in best_ckpt cannot change the dense channel ?

Sorry to bother the author's time !

[yxlu-0102]

The checkpoint file in best_ckpt is the pre-trained best checkpoint with the default dense channel, which is unchangeable. You should use your self-trained checkpoint for inference.