When the GPU is occupied

[MaNing1924382115]

run evaluation.py
When the GPU is occupied, for example, how big the GPU needs to be in 20 seconds

[SherifAbdulatif]

You mean if you give a 20 seconds track what is the GPU utilization in inference?

[daniel-nocap]

I am experiencing CUDA out of memory errors on a GPU with 40gb of memory whenever I perform inference on an audio file longer than 10 seconds. There is a massive spike in GPU memory utilization at this line in evaluation.py: est_real, est_imag = model(noisy_spec)

Any suggestions for how to reduce peak memory usage would be greatly appreciated. Thank you

[ruizhecao96]

I am experiencing CUDA out of memory errors on a GPU with 40gb of memory whenever I perform inference on an audio file longer than 10 seconds. There is a massive spike in GPU memory utilization at this line in evaluation.py: est_real, est_imag = model(noisy_spec)

Any suggestions for how to reduce peak memory usage would be greatly appreciated. Thank you

Just reshape the input, for example you have 20 seconds track, then reshape it from (1, 1600020) to (2, 1600010) or (4, 16000*5), and reshape it back in the output. Since the computational complexity increase quadratically with the increase of the input length.