Open vaynelau opened 4 years ago
When I trained the base model, I observed that the memory usage continued to rise and finally reached the maximum limit and the process was killed. My RAM is 250GB and I used four 16GB Tesla V100 GPU. I tried to decrease the batch size but it did not make any difference. I wonder how much memory you used during the base training and if there is something wrong I did. Thank you.
Hello, I meet this problem too. Have you solved it?
decrease the batch size if you do not have a high end GPU
When I trained the base model, I observed that the memory usage continued to rise and finally reached the maximum limit and the process was killed. My RAM is 250GB and I used four 16GB Tesla V100 GPU. I tried to decrease the batch size but it did not make any difference. I wonder how much memory you used during the base training and if there is something wrong I did. Thank you.