Open superlwx opened 4 years ago
why gpu memory increases suddenly resulting out of memory in training? 6000MB->10000MB in random epoch
I'm not sure whether the memory increasement happened before. Our used machine has 24GB GPU memory. We have not encountered the out of memory problem during training.
why gpu memory increases suddenly resulting out of memory in training? 6000MB->10000MB in random epoch