I use one RTX3090 to train neuman,and it takes about 500 hours to finish 1000000 iters,one iter will take 2 seconds,and I find although the memory used by GPU is high, but the usage rate (percentage) of memory is not high,rarely exceeding 50%,often below 10%,how should I deal with it?
I use one RTX3090 to train neuman,and it takes about 500 hours to finish 1000000 iters,one iter will take 2 seconds,and I find although the memory used by GPU is high, but the usage rate (percentage) of memory is not high,rarely exceeding 50%,often below 10%,how should I deal with it?